Date   

bpf: Failed to load program: Permission denied

Jacob Steadman
 

Hi,

I'm new to BPF. I'm trying to write a program that analyses the structure o= f DNS requests. I keep getting the following error (bellow) at a certain point in the code (bellow).

The error only occurs when I try to "return -1;" (i.e. allow the packet). I= f I remove this line the program executes as expected.

I wonder if it could be an issue with the kernel version rather than the co= de? (Ubuntu 16.04.4 LTS, Kernel version 4.4.0-87-generic)

Error***************************************
bpf: Failed to load program: Permission denied
...
...
R2 invalid mem access 'inv'

HINT: The invalid mem access 'inv' error can happen if you try to dereferen= ce memory without first using bpf_probe_read() to copy it to the BPF stack.=  Sometimes the bpf_probe_read is automatic by the bcc rewriter, other times=  you'll need to be explicit.

Traceback (most recent call last):
  File "dns_matching.py", line 57, in <module>
    function_dns_matching =3D bpf.load_func("dns_exfil_detection_v2", BPF.S=
OCKET_FILTER)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 379, in loa= d_func
    (func_name, errstr))
Exception: Failed to load BPF program dns_exfil_detection_v2: Permission de= nied
********************************************


Code****************************************
        #pragma unroll
        for(i =3D 0; i<255;i++){
                c =3D cursor_advance(cursor, 1);

                if (c->c =3D=3D 0)
                                break;

key.p[i] =3D c->c;

                //**ensure this is the correct max length of a subdomain**
                if(c->c < 0x0f){
                                subdomLengths[subdomainCount] =3D (u16) c->= c;
                                subdomainCount =3D subdomainCount +1;
                 }
        }

*** if(subdomLengths[subdomainCount] =3D=3D 2 && subdomLengths[subdomainCou= nt-1]  =3D=3D2 && subdomainCount <4 ){
***         return -1;
*** }
*********************************************


Re: R? min value is negative, either use unsigned or 'var &= const' #verifier

Simon
 

Here a python 2 version.


Re: R? min value is negative, either use unsigned or 'var &= const' #verifier

Yonghong Song
 

On Mon, Mar 11, 2019 at 4:08 AM Simon <contact@...> wrote:

I tried to understand again this verifier error again and probably my previous post does not contain enough information.

I understand that :

93: (67) r0 <<= 32
294: (c7) r0 s>>= 32
295: (b7) r1 = 0
296: (b7) r2 = 0
297: (bf) r3 = r8
298: (79) r4 = *(u64 *)(r10 -40)
299: (bf) r5 = r0
300: (85) call bpf_csum_diff#28
R4 min value is negative, either use unsigned or 'var &= const'

is about this line (in ipv4_l4_csum)

*csum = bpf_csum_diff(0, 0, data_start, data_size, *csum);

R1=0,
R2=0,
R3= R8=pkt(id=0,off=34,r=42,imm=0) = data_start = a pointer to struct udphdr *udp
R4= something in the stack = data_size = __u16 udp_len

So I can not understand how this bring to R4 min value is negative, either use unsigned or 'var &= const'
I took a brief look. Indeed, it is very strange. I can see proper
value of udp_len is stored into
r10 - 40, but when it is retrieved later, the value became unkown....

I will try to experiment with this problem later this week.

Could you do me a favor to make it reproducible with python2? My env.
flexible to rebuild/retry with kernel is python2 friendly.



298: (79) r4 = *(u64 *)(r10 -40)

As I understand this line, r4 will get a value in the stack (R10=fp0,call_-1 fp-48=pkt) and cast this value in a u64, so unsigned. (min value = 0)

(By the way I can not understand why this is a u64 and not a u16 as udp_len variable or u32 as data_size parameter of ipv4_l4_csum function or u32 as tosize from bpf_csum_diff function...)

I tried to use the &= tricks like :

data_size = data_size & 0x1ff;
*csum = bpf_csum_diff(0, 0, data_start, data_size, *csum);

Same issue ...

Here a more longer trace from the verifier :

R0=inv(id=0,umax_value=4295032831,var_off=(0x0; 0x1ffffffff))
R1=inv(id=0,umax_value=65536,var_off=(0x0; 0x1ffff))
R6=ctx(id=0,off=0,imm=0)
R7=pkt(id=0,off=0,r=42,imm=0)
R8=pkt(id=0,off=34,r=42,imm=0)
R9=pkt(id=0,off=30,r=42,imm=0)
R10=fp0,call_-1 fp-48=pkt
239: (57) r0 &= 65535
240: (0f) r0 += r1
241: (bf) r1 = r0
242: (77) r1 >>= 16
243: (0f) r1 += r0
244: (a7) r1 ^= -1
245: (6b) *(u16 *)(r7 +24) = r1
246: (b7) r1 = 0
247: (6b) *(u16 *)(r7 +40) = r1
248: (b7) r1 = 0
249: (b7) r2 = 0
250: (79) r3 = *(u64 *)(r10 -48)
251: (b7) r4 = 4
252: (b7) r5 = 0
253: (85) call bpf_csum_diff#28
254: (67) r0 <<= 32
255: (c7) r0 s>>= 32
256: (b7) r1 = 0
257: (b7) r2 = 0
258: (bf) r3 = r9
259: (b7) r4 = 4
260: (bf) r5 = r0
261: (85) call bpf_csum_diff#28
262: (71) r1 = *(u8 *)(r7 +23)
263: (dc) r1 = be32 r1
264: (63) *(u32 *)(r10 -24) = r1
265: (67) r0 <<= 32
266: (c7) r0 s>>= 32
267: (bf) r9 = r10
268: (07) r9 += -24
269: (b7) r1 = 0
270: (b7) r2 = 0
271: (bf) r3 = r9
272: (b7) r4 = 4
273: (bf) r5 = r0
274: (85) call bpf_csum_diff#28
275: (79) r1 = *(u64 *)(r10 -40)
276: (dc) r1 = be32 r1
277: (63) *(u32 *)(r10 -24) = r1
278: (67) r0 <<= 32
279: (c7) r0 s>>= 32
280: (b7) r1 = 0
281: (b7) r2 = 0
282: (bf) r3 = r9
283: (b7) r4 = 4
284: (bf) r5 = r0
285: (85) call bpf_csum_diff#28
286: (67) r0 <<= 32
287: (c7) r0 s>>= 32
288: (b7) r1 = 0
289: (b7) r2 = 0
290: (bf) r3 = r8
291: (79) r4 = *(u64 *)(r10 -40)
292: (bf) r5 = r0
293: (85) call bpf_csum_diff#28

I reference the commit instead of repository to keep the link consistent over the time : https://github.com/sbernard31/udploadbalancer/tree/5ca93d0893a60bc70a75f30eb5cfde496a9e5d93

Again do not hesitate to redirect me to better place if I'm not asking at the right place :)

Thx again for your time.


Re: R? min value is negative, either use unsigned or 'var &= const' #verifier

Simon
 

I tried to understand again this verifier error again and probably my previous post does not contain enough information.

I  understand that :

93: (67) r0 <<= 32
294: (c7) r0 s>>= 32
295: (b7) r1 = 0
296: (b7) r2 = 0
297: (bf) r3 = r8
298: (79) r4 = *(u64 *)(r10 -40)
299: (bf) r5 = r0
300: (85) call bpf_csum_diff#28
R4 min value is negative, either use unsigned or 'var &= const'

is about this line  (in ipv4_l4_csum)

  *csum = bpf_csum_diff(0, 0, data_start, data_size, *csum);

R1=0,
R2=0,
R3= R8=pkt(id=0,off=34,r=42,imm=0) = data_start =  a pointer to struct udphdr *udp
R4= something in the stack  = data_size = __u16 udp_len

So I can not understand how this bring to R4 min value is negative, either use unsigned or 'var &= const'

298: (79) r4 = *(u64 *)(r10 -40)

As I understand this line, r4 will get a value in the stack (R10=fp0,call_-1 fp-48=pkt) and cast  this value in a u64, so unsigned. (min value = 0)

(By the way I can not understand why this is a u64 and not a u16 as udp_len variable or u32 as  data_size parameter of ipv4_l4_csum function or u32 as tosize from bpf_csum_diff function...)

I tried to use the &= tricks like :

data_size = data_size & 0x1ff;
*csum = bpf_csum_diff(0, 0, data_start, data_size, *csum);

Same issue ...

Here a more longer trace from the verifier :

R0=inv(id=0,umax_value=4295032831,var_off=(0x0; 0x1ffffffff)) 
R1=inv(id=0,umax_value=65536,var_off=(0x0; 0x1ffff))
R6=ctx(id=0,off=0,imm=0)
R7=pkt(id=0,off=0,r=42,imm=0)
R8=pkt(id=0,off=34,r=42,imm=0)
R9=pkt(id=0,off=30,r=42,imm=0)
R10=fp0,call_-1 fp-48=pkt 239: (57) r0 &= 65535 240: (0f) r0 += r1 241: (bf) r1 = r0 242: (77) r1 >>= 16 243: (0f) r1 += r0 244: (a7) r1 ^= -1 245: (6b) *(u16 *)(r7 +24) = r1 246: (b7) r1 = 0 247: (6b) *(u16 *)(r7 +40) = r1 248: (b7) r1 = 0 249: (b7) r2 = 0 250: (79) r3 = *(u64 *)(r10 -48) 251: (b7) r4 = 4 252: (b7) r5 = 0 253: (85) call bpf_csum_diff#28 254: (67) r0 <<= 32 255: (c7) r0 s>>= 32 256: (b7) r1 = 0 257: (b7) r2 = 0 258: (bf) r3 = r9 259: (b7) r4 = 4 260: (bf) r5 = r0 261: (85) call bpf_csum_diff#28 262: (71) r1 = *(u8 *)(r7 +23) 263: (dc) r1 = be32 r1 264: (63) *(u32 *)(r10 -24) = r1 265: (67) r0 <<= 32 266: (c7) r0 s>>= 32 267: (bf) r9 = r10 268: (07) r9 += -24 269: (b7) r1 = 0 270: (b7) r2 = 0 271: (bf) r3 = r9 272: (b7) r4 = 4 273: (bf) r5 = r0 274: (85) call bpf_csum_diff#28 275: (79) r1 = *(u64 *)(r10 -40) 276: (dc) r1 = be32 r1 277: (63) *(u32 *)(r10 -24) = r1 278: (67) r0 <<= 32 279: (c7) r0 s>>= 32 280: (b7) r1 = 0 281: (b7) r2 = 0 282: (bf) r3 = r9 283: (b7) r4 = 4 284: (bf) r5 = r0 285: (85) call bpf_csum_diff#28 286: (67) r0 <<= 32 287: (c7) r0 s>>= 32 288: (b7) r1 = 0 289: (b7) r2 = 0 290: (bf) r3 = r8 291: (79) r4 = *(u64 *)(r10 -40) 292: (bf) r5 = r0 293: (85) call bpf_csum_diff#28

I reference the commit instead of repository to keep the link consistent over the time : https://github.com/sbernard31/udploadbalancer/tree/5ca93d0893a60bc70a75f30eb5cfde496a9e5d93

Again do not hesitate to redirect me to better place if I'm not asking at the right place :)

Thx again for your time.


Re: math between pkt pointer and register with unbounded min value is not allowed #verifier

Yonghong Song
 

On Fri, Mar 8, 2019 at 9:22 AM Simon <contact@...> wrote:


35: (69) r3 = *(u16 *)(r7 +38)
36: (dc) r3 = be16 r3

r3 get the value from memory, its value could be any one as permitted
by the type.

Does it mean that r3 is considered as be16 ? I do not understand why as I explicitly convert it in u16.
The be16 is to convert r3 with big endian encoding. If the host system
is big endian, it will do nothing. Otherwise,
it will convert from little endian to big endian.


This output language is a readable format of bpf bytecode, right ? Is there any documentation to lean/understand it ?
Yes, there is no documentation. It intends to be self explanatory. I
guess "be16" is special and may need some documentation. Otherwise
assembly-style codes should be easy to understand.


The compiler does the right thing, just verifier is not advanced enough.

Is it worthy to share this issue of verifier.c with bpf maintainers ? The compiler which is used here is clang which is called by bcc, right ?
I am also a regular kernel/bpf reviewer. The bpf maintainers/community
are aware of this limitation. As you mentioned, the verifier is
already very complex. To implement complex tracking like described in
this thread will make verifier even more complex, hence this is
delayed. One of reason is that we have reasonable, although
unpleasant, workarounds.

Yes, it is compiled with clang.


Yes, you will need some source workaround. You could try below (untested):
+ udp_len = udp_len & 0x1ff;

I tested it and it seems to work. Thx a lot !!

But that means I can not use the u16 max value ?
You can. I add that because you have a test to limit the range of the
value to 511.



Re: minutes: IO Visor TSC/Dev Meeting

Saeed Mahameed
 

Hi Guys,

My agenda for next meeting:

1) unifying and centralizing XDP statistics accounting [1].
2) XDP resource management, User API [2].
3) XDP meta data via btf (in kernel BTF registration).
4) all of the above issues share one common problem, which is the lack
of a unified user interface
without it, We really can't make a real progress.

I just sent a proposal [3] for away to achieve the unified interface,
please look it up and let me know your thoughts.
[1] https://github.com/xdp-project/xdp-project/blob/master/xdp-project.org#statistics-per-xdp-action
[2] https://github.com/xdp-project/xdp-project/blob/master/xdp-project.org#better-ndo_xdp_xmit-resource-management
[3] Subject: "[RFC][Proposal] BPF Control MAP"

Thanks,
Saeed

On Wed, Mar 6, 2019 at 12:51 PM Brenden Blanco <bblanco@...> wrote:

Hi All,

Thank you for joining the call today. Here are my notes from the discussion.

Thanks,
Brenden

=== Discussion ===
Brenden:
* Plan to tag release to coincide with kernel 5.0

Brendan:
* Speaking this weekend at SCaLE in Los Angeles

Yonghong:
* LLVM work
* compile once - run anywhere WIP
* support for static variables

Daniel:
* Global data support work in kernel continues
* Ability to lock maps as read-only
* bugfixes after merge window

Alexei:
* Some thoughts on future work of BPF
* especially with introduction of BTF
* overall needs concerted effort to improve debuggability
* BTF for programs itself with source/type/layout information
* structures for maps and global data
* suggest to always require type information
(already turned on by default in bcc and supported by llvm)
* Some extra hoops to jump through for driver embedded BPF
* to be enabled with a sysctl
* kernel support is ready
* some long tail of support - e.g. systemd has raw assembly BPF
* kconfig option - eventual deprecation
* if kernel is default strict, llvm should automatically emit BTF as well
* memcg accounting patch status?
* Daniel - still being worked on
* proposal to enable the same accounting for verifier memory
* helps to enable verifier multithreading

Jakub:
* question regarding global data atomicity
* Daniel - requires read once / write once instructions to work properly
* some todo work on documentation, interpreter + jit implementations
* depends on architecture (machine word size guarantees only)

Jesper:
* which llvm release supports BTF
* landed in December - will be in 8.0, better in 9.0
* working on tutorial for xdp at netdev
* https://www.netdevconf.org/0x13/session.html?tutorial-XDP-hands-on
* soliciting feedback
* https://github.com/xdp-project/xdp-tutorial/

Saeed:
* request to devote some time in the next meeting to iron out some XDP issues
* please send an agend in reply to the reminder email before next call
* prepare discussion over email in between time

=== Attendees ===
Alexei Starovoitov
Marco Leogrande
Mauricio Vasquez
Paul Chaignon
Brenden Blanco
Jiong Wang
Yonghong Song
Daniel Borkmann
Jesper Brouer
Quentin Monnet
Dan Siemon
Jakub Kicinski
Saeed
John
Yutaro



R? min value is negative, either use unsigned or 'var &= const' #verifier

Simon
 

I suspect I fall in another issue with verifier.

This code (taking from katran : I just changed data_size type to __u32):

__attribute__((__always_inline__))
static inline void ipv4_l4_csum(void *data_start, __u32 data_size,
                                __u64 *csum, struct iphdr *iph) {
  __u32 tmp = 0;
  *csum = bpf_csum_diff(0, 0, &iph->saddr, sizeof(__be32), *csum);
  *csum = bpf_csum_diff(0, 0, &iph->daddr, sizeof(__be32), *csum);
  tmp = __builtin_bswap32((__u32)(iph->protocol));
  *csum = bpf_csum_diff(0, 0, &tmp, sizeof(__u32), *csum);
  tmp = __builtin_bswap32((__u32)(data_size));
  *csum = bpf_csum_diff(0, 0, &tmp, sizeof(__u32), *csum);
  *csum = bpf_csum_diff(0, 0, data_start, data_size, *csum);
  *csum = csum_fold_helper(*csum);
}
But it brings to this verifier issue :
93: (67) r0 <<= 32
294: (c7) r0 s>>= 32
295: (b7) r1 = 0
296: (b7) r2 = 0
297: (bf) r3 = r8
298: (79) r4 = *(u64 *)(r10 -40)
299: (bf) r5 = r0
300: (85) call bpf_csum_diff#28
R4 min value is negative, either use unsigned or 'var &= const'
The whole code is available here.
This is maybe relative to my previous issue.


Re: math between pkt pointer and register with unbounded min value is not allowed #verifier

Simon
 


35: (69) r3 = *(u16 *)(r7 +38)
36: (dc) r3 = be16 r3
r3 get the value from memory, its value could be any one as permitted
by the type.

Does it mean that r3 is considered as be16 ? I do not understand why as I explicitly convert it in u16.

This output language is a readable format of bpf bytecode, right ? Is there any documentation to lean/understand it ? 

The compiler does the right thing, just verifier is not advanced enough.
Is it worthy to share this issue of verifier.c with bpf maintainers ? The compiler which is used here is clang which is called by bcc, right ?

Yes, you will need some source workaround. You could try below (untested):
+ udp_len = udp_len & 0x1ff;

I tested it and it seems to work. Thx a lot !!

But that means I can not use the u16 max value ?


Re: math between pkt pointer and register with unbounded min value is not allowed #verifier

Jiong Wang
 

Yonghong Song writes:

On Wed, Mar 6, 2019 at 7:08 AM <contact@...> wrote:

I'm playing with bcc to prototype an UDP load balancer.

I'm facing an issue that I didn't succeed to understand...

In my code I tried to validate my UDP packet using code like this :

struct udphdr *udp;
udp = iph + 1;
if (udp + 1 > data_end)
return XDP_DROP;
__u16 udp_len = bpf_ntohs(udp->len);
//__u16 udp_len = 8;
if (udp_len < 8)
return XDP_DROP;
if (udp_len > 512) // TODO use a more approriate max value
return XDP_DROP;
if ((void *) udp + udp_len > data_end)
return XDP_DROP;

And the verifier does not like it ..
This is caused by compiler optimizations.


28: (71) r2 = *(u8 *)(r7 +23)
29: (b7) r0 = 2
30: (55) if r2 != 0x11 goto pc+334
R0=inv2 R1=pkt_end(id=0,off=0,imm=0) R2=inv17 R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=34,imm=0) R8=pkt(id=0,off=34,r=34,imm=0) R9=pkt(id=0,off=14,r=34,imm=0) R10=fp0,call_-1
31: (bf) r2 = r8
32: (07) r2 += 8
33: (b7) r0 = 1
34: (2d) if r2 > r1 goto pc+330
R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=pkt(id=0,off=42,r=42,imm=0) R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1
35: (69) r3 = *(u16 *)(r7 +38)
36: (dc) r3 = be16 r3
r3 get the value from memory, its value could be any one as permitted
by the type.

37: (bf) r2 = r3
38: (07) r2 += -8
39: (57) r2 &= 65535
40: (b7) r0 = 1
41: (25) if r2 > 0x1f8 goto pc+323
test is done by r2. We indeed get better range for r2 (below:
R2=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) )
but r3 range is not tightened.
I had run into similar issue when debugging some other rejection before
JMP32 introduced when LLVM was generating similar sequences under defult
64-bit mode, but IIRC LLVM generates betweer sequences with -mattr=alu32,
under which it will just use w3 (as the type should be optimized into
32-bit) for the comparison.

So, I guess this testcase could have easier sequence for verifier under
ALU32 mode. But for this case, BPF_END is used which doesn't have
sub-register code-gen support inside LLVM for be16 and be32 at the moment
(noticed this several days ago when doing some other benchmarking).

If I have .i file, I could do a quick prototype to see if ALU32 could
improve this.

Regards,
Jiong


Re: math between pkt pointer and register with unbounded min value is not allowed #verifier

Yonghong Song
 

On Wed, Mar 6, 2019 at 7:08 AM <contact@...> wrote:

I'm playing with bcc to prototype an UDP load balancer.

I'm facing an issue that I didn't succeed to understand...

In my code I tried to validate my UDP packet using code like this :

struct udphdr *udp;
udp = iph + 1;
if (udp + 1 > data_end)
return XDP_DROP;
__u16 udp_len = bpf_ntohs(udp->len);
//__u16 udp_len = 8;
if (udp_len < 8)
return XDP_DROP;
if (udp_len > 512) // TODO use a more approriate max value
return XDP_DROP;
if ((void *) udp + udp_len > data_end)
return XDP_DROP;

And the verifier does not like it ..
This is caused by compiler optimizations.


28: (71) r2 = *(u8 *)(r7 +23)
29: (b7) r0 = 2
30: (55) if r2 != 0x11 goto pc+334
R0=inv2 R1=pkt_end(id=0,off=0,imm=0) R2=inv17 R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=34,imm=0) R8=pkt(id=0,off=34,r=34,imm=0) R9=pkt(id=0,off=14,r=34,imm=0) R10=fp0,call_-1
31: (bf) r2 = r8
32: (07) r2 += 8
33: (b7) r0 = 1
34: (2d) if r2 > r1 goto pc+330
R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=pkt(id=0,off=42,r=42,imm=0) R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1
35: (69) r3 = *(u16 *)(r7 +38)
36: (dc) r3 = be16 r3
r3 get the value from memory, its value could be any one as permitted
by the type.

37: (bf) r2 = r3
38: (07) r2 += -8
39: (57) r2 &= 65535
40: (b7) r0 = 1
41: (25) if r2 > 0x1f8 goto pc+323
test is done by r2. We indeed get better range for r2 (below:
R2=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) )
but r3 range is not tightened.

R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) R3=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1
42: (bf) r2 = r7
43: (0f) r2 += r3
math between pkt pointer and register with unbounded min value is not allowed
Here, we use r3 to do arith and the verifier not happy. If we use the
tightened old r2, it may work.
The compiler does the right thing, just verifier is not advanced enough.


I'm pretty sure the issue is about udp_len, that's why I tried to validate its value before to use it ... but without success...
When I set udp_len to 8 (just for testing) this seems to works. Any idea about that ?
Yes, you will need some source workaround. You could try below (untested):

if (udp_len > 512) // TODO use a more approriate max value
return XDP_DROP;
+ udp_len = udp_len & 0x1ff;
if ((void *) udp + udp_len > data_end)
return XDP_DROP;


Full code is available here : https://gist.github.com/sbernard31/d4fee7518a1ff130452211c0d355b3f7

(I'm using python-bpfcc 0.8.0-4 from debian sid with a 4.19.12 kernel)
(I don't know if this is the right place for this kind of question, )


Re: Sharing eBPF map between two eBPF kernel programs

Yonghong Song
 

On Mon, Mar 4, 2019 at 10:45 PM Kanthi P <pavuluri.kanthi@...> wrote:

Thanks for that!

I think those work because bcc has given utilities to do that.

I am using C and we use libbpf to load programs and to do map operations.

Do you know if we can achieve it in this way too?
You should be able to do the work with C as well. Just use APIs from libbpf.h,
e.g., bpf_prog_get_next_id, bpf_prog_get_fd_by_id, bpf_map_get_fd_by_id.
Yes, we may miss bpf_map_get_next_id and bpf_obj_get_info_by_fd.
You can just add prototype to bcc/src/cc/libbpf.h and the implementation
is libbpf submodule.

We can add the missing piece in bcc/src/cc/{libbpf.h, libbpf.c}.


Thanks,
Kanthi

On Tue, Mar 5, 2019 at 2:21 AM Y Song <ys114321@...> wrote:

On Mon, Mar 4, 2019 at 9:52 AM Kanthi P <Pavuluri.kanthi@...> wrote:

Hi,

I have two eBPF programs, each with its own user and kernel programs.

One of the programs defines an eBPF map.

How do I access this map from another eBPF kernel program?
In examples/cpp, we have UseExternalMap.cc, which gives an example of
how to share maps between two processes with C++ interface.
Basically, if you know the ID of the map in another application (you
can see id use bpftool), you can get a fd for that map by ID with API
`bpf_map_get_fd_by_id()`, and example shows how to inject the new `fd`
to bcc compilation system.

We do not have a python way to share map yet bwtween two different processes.

Please take a look. If you feel the interface needs to improve to
support your use case, we can do that.


PS:

I can pin the map and that helps me access this map from other program's user space.
But looking for a direct way in which I can access this map from other programs's kernel space.

Thanks!


minutes: IO Visor TSC/Dev Meeting

Brenden Blanco
 

Hi All,

Thank you for joining the call today. Here are my notes from the discussion.

Thanks,
Brenden

=== Discussion ===
Brenden:
* Plan to tag release to coincide with kernel 5.0

Brendan:
* Speaking this weekend at SCaLE in Los Angeles

Yonghong:
* LLVM work
* compile once - run anywhere WIP
* support for static variables

Daniel:
* Global data support work in kernel continues
* Ability to lock maps as read-only
* bugfixes after merge window

Alexei:
* Some thoughts on future work of BPF
* especially with introduction of BTF
* overall needs concerted effort to improve debuggability
* BTF for programs itself with source/type/layout information
* structures for maps and global data
* suggest to always require type information
(already turned on by default in bcc and supported by llvm)
* Some extra hoops to jump through for driver embedded BPF
* to be enabled with a sysctl
* kernel support is ready
* some long tail of support - e.g. systemd has raw assembly BPF
* kconfig option - eventual deprecation
* if kernel is default strict, llvm should automatically emit BTF as well
* memcg accounting patch status?
* Daniel - still being worked on
* proposal to enable the same accounting for verifier memory
* helps to enable verifier multithreading

Jakub:
* question regarding global data atomicity
* Daniel - requires read once / write once instructions to work properly
* some todo work on documentation, interpreter + jit implementations
* depends on architecture (machine word size guarantees only)

Jesper:
* which llvm release supports BTF
* landed in December - will be in 8.0, better in 9.0
* working on tutorial for xdp at netdev
* https://www.netdevconf.org/0x13/session.html?tutorial-XDP-hands-on
* soliciting feedback
* https://github.com/xdp-project/xdp-tutorial/

Saeed:
* request to devote some time in the next meeting to iron out some XDP issues
* please send an agend in reply to the reminder email before next call
* prepare discussion over email in between time

=== Attendees ===
Alexei Starovoitov
Marco Leogrande
Mauricio Vasquez
Paul Chaignon
Brenden Blanco
Jiong Wang
Yonghong Song
Daniel Borkmann
Jesper Brouer
Quentin Monnet
Dan Siemon
Jakub Kicinski
Saeed
John
Yutaro


math between pkt pointer and register with unbounded min value is not allowed #verifier

Simon
 

I'm playing with bcc to prototype an UDP load balancer.

I'm facing an issue that I didn't succeed to understand...

In my code I tried to validate my UDP packet using code like this :

    struct udphdr *udp;
    udp = iph + 1;
    if (udp + 1 > data_end)
        return XDP_DROP;
    __u16 udp_len = bpf_ntohs(udp->len);
    //__u16 udp_len = 8;
    if (udp_len < 8)
        return XDP_DROP;
    if (udp_len > 512) // TODO use a more approriate max value
        return XDP_DROP;
    if ((void *) udp + udp_len > data_end)
        return XDP_DROP;
And the verifier does not like it ..
28: (71) r2 = *(u8 *)(r7 +23)
29: (b7) r0 = 2
30: (55) if r2 != 0x11 goto pc+334
 R0=inv2 R1=pkt_end(id=0,off=0,imm=0) R2=inv17 R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=34,imm=0) R8=pkt(id=0,off=34,r=34,imm=0) R9=pkt(id=0,off=14,r=34,imm=0) R10=fp0,call_-1
31: (bf) r2 = r8
32: (07) r2 += 8
33: (b7) r0 = 1
34: (2d) if r2 > r1 goto pc+330
 R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=pkt(id=0,off=42,r=42,imm=0) R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1
35: (69) r3 = *(u16 *)(r7 +38)
36: (dc) r3 = be16 r3
37: (bf) r2 = r3
38: (07) r2 += -8
39: (57) r2 &= 65535
40: (b7) r0 = 1
41: (25) if r2 > 0x1f8 goto pc+323
 R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) R3=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1
42: (bf) r2 = r7
43: (0f) r2 += r3
math between pkt pointer and register with unbounded min value is not allowed

I'm pretty sure the issue is about udp_len, that's why I tried to validate its value before to use it ... but without success...
When I set udp_len to 8 (just for testing) this seems to works. Any idea about that ?

Full code is available here : https://gist.github.com/sbernard31/d4fee7518a1ff130452211c0d355b3f7

(I'm using python-bpfcc  0.8.0-4 from debian sid with a 4.19.12 kernel)
(I don't know if this is the right place for this kind of question, )


reminder: IO Visor TSC/Dev Meeting

Brenden Blanco
 

Please join us tomorrow for our bi-weekly call. As usual, this meeting is
open to everybody and completely optional.
You might be interested to join if:
You want to know what is going on in BPF land
You are doing something interesting yourself with BPF and would like to share
You want to know what the heck BPF is

=== IO Visor Dev/TSC Meeting ===

Every 2 weeks on Wednesday, from Wednesday, January 25, 2017, to no end date
11:00 am | Pacific Daylight Time (San Francisco, GMT-07:00) | 30 min

https://bluejeans.com/568677804/

https://www.timeanddate.com/worldclock/meetingdetails.html?year=2019&month=3&day=6&hour=19&min=0&sec=0&p1=900


Re: Sharing eBPF map between two eBPF kernel programs

Kanthi P <Pavuluri.kanthi@...>
 

Thanks for that!

I think those work because bcc has given utilities to do that.

I am using C and we use libbpf to load programs and to do map operations. 

Do you know if we can achieve it in this way too?

Thanks,
Kanthi

On Tue, Mar 5, 2019 at 2:21 AM Y Song <ys114321@...> wrote:
On Mon, Mar 4, 2019 at 9:52 AM Kanthi P <Pavuluri.kanthi@...> wrote:
>
> Hi,
>
> I have two eBPF programs, each with its own user and kernel programs.
>
> One of the programs defines an eBPF map.
>
> How do I access this map from another eBPF kernel program?

In examples/cpp, we have UseExternalMap.cc, which gives an example of
how to share maps between two processes with C++ interface.
Basically, if you know the ID of the map in another application (you
can see id use bpftool), you can get a fd for that map by ID with API
`bpf_map_get_fd_by_id()`, and example shows how to inject the new `fd`
to bcc compilation system.

We do not have a python way to share map yet bwtween two different processes.

Please take a look. If you feel the interface needs to improve to
support your use case, we can do that.

>
> PS:
>
> I can pin the map and that helps me access this map from other program's user space.
> But looking for a direct way in which I can access this map from other programs's kernel space.
>
> Thanks!


Re: Sharing eBPF map between two eBPF kernel programs

Yonghong Song
 

On Mon, Mar 4, 2019 at 9:52 AM Kanthi P <Pavuluri.kanthi@...> wrote:

Hi,

I have two eBPF programs, each with its own user and kernel programs.

One of the programs defines an eBPF map.

How do I access this map from another eBPF kernel program?
In examples/cpp, we have UseExternalMap.cc, which gives an example of
how to share maps between two processes with C++ interface.
Basically, if you know the ID of the map in another application (you
can see id use bpftool), you can get a fd for that map by ID with API
`bpf_map_get_fd_by_id()`, and example shows how to inject the new `fd`
to bcc compilation system.

We do not have a python way to share map yet bwtween two different processes.

Please take a look. If you feel the interface needs to improve to
support your use case, we can do that.


PS:

I can pin the map and that helps me access this map from other program's user space.
But looking for a direct way in which I can access this map from other programs's kernel space.

Thanks!


Sharing eBPF map between two eBPF kernel programs

Kanthi P <Pavuluri.kanthi@...>
 

Hi,

I have two eBPF programs, each with its own user and kernel programs.

One of the programs defines an eBPF map.

How do I access this map from another eBPF kernel program?

PS:

I can pin the map and that helps me access this map from other program's user space.
But looking for a direct way in which I can access this map from other programs's kernel space.

Thanks!


about kprobe/kretprobe matching

ovinee@...
 

Hi,

I'm trying to access the packet content in tcp_transmit_skb using eBPF/BCC.
I defined kprobe____tcp_transmit_skb/kretprobe____tcp_transmit_skb functions.
Since in kretprobe we cannot access the function arguments (I'm not sure), I stored skb(struct sk_buff *) in kprobe____tcp_transmit_skb by using the pid as the key.

u64 pid = bpf_get_current_pid_tgid();
curr_skb.update(&pid, &skb);

However, the issue is that tcp_transmit_skb function can be called multiple times before kretprobe____tcp_transmit_skb is executed with the same pid.
So I cannot correctly match between kprobe____tcp_transmit_skb and kretprobe____tcp_transmit_skb functions for the same skb.
Is there any good way to match between kprobe and kretprobe for the same function call?
Or is there any method that I can access the function arguments in kretprobe?

Thanks,
Youngbin


share maps for two prog

Иван Иванов
 

I have two port running xdp prog. Each have same type, struct and purpose maps. I want to use shared map. So tested such a way, run xdp prog on each device with only one func - bpf_tail_call. This calls to the third xdp prog and gets and saves data to its maps. So is it only such a way, or may be another way? 
If only one, how can i control, that bpf_tail_call not drop some packets?
And what "cost" have bpf_tail_call? Is it cheap or not? Or its better to send to main(third) prog to save data in man not xdp_md , but may by some (8 bytes) needed to save data?


Re: XDP Offloading

Andrew Gospodarek
 

On Mon, Feb 25, 2019 at 08:50:52AM -0800, mdimolianis@... wrote:
Hi all,
I am wondering except for Netronome Cards if there are any other NICs that support XDP Offloading?
Thank you in advance!
Many of the ARM-based SmartNICs[1] allow users to download and execute
code on the cards themselves to save the server cores from being used
for excuting the XDP programs.

The Broadcom Stingray/PS225[2] is one of those cards.

This is possible as the NIC cards themselves are also running full Linux
distros with console access so you can treat it as a server inside your
server.

1. I define a SmartNIC NIC where control and dataplane traffic can be
offload from server cores to a NIC with general purpose ARM cores
running Linux.

2. https://www.broadcom.com/products/ethernet-connectivity/smartnic/ps225

421 - 440 of 2027