Date   

R? min value is negative, either use unsigned or 'var &= const' #verifier

Simon
 

I suspect I fall in another issue with verifier.

This code (taking from katran : I just changed data_size type to __u32):

__attribute__((__always_inline__))
static inline void ipv4_l4_csum(void *data_start, __u32 data_size,
                                __u64 *csum, struct iphdr *iph) {
  __u32 tmp = 0;
  *csum = bpf_csum_diff(0, 0, &iph->saddr, sizeof(__be32), *csum);
  *csum = bpf_csum_diff(0, 0, &iph->daddr, sizeof(__be32), *csum);
  tmp = __builtin_bswap32((__u32)(iph->protocol));
  *csum = bpf_csum_diff(0, 0, &tmp, sizeof(__u32), *csum);
  tmp = __builtin_bswap32((__u32)(data_size));
  *csum = bpf_csum_diff(0, 0, &tmp, sizeof(__u32), *csum);
  *csum = bpf_csum_diff(0, 0, data_start, data_size, *csum);
  *csum = csum_fold_helper(*csum);
}
But it brings to this verifier issue :
93: (67) r0 <<= 32
294: (c7) r0 s>>= 32
295: (b7) r1 = 0
296: (b7) r2 = 0
297: (bf) r3 = r8
298: (79) r4 = *(u64 *)(r10 -40)
299: (bf) r5 = r0
300: (85) call bpf_csum_diff#28
R4 min value is negative, either use unsigned or 'var &= const'
The whole code is available here.
This is maybe relative to my previous issue.


Re: math between pkt pointer and register with unbounded min value is not allowed #verifier

Simon
 


35: (69) r3 = *(u16 *)(r7 +38)
36: (dc) r3 = be16 r3
r3 get the value from memory, its value could be any one as permitted
by the type.

Does it mean that r3 is considered as be16 ? I do not understand why as I explicitly convert it in u16.

This output language is a readable format of bpf bytecode, right ? Is there any documentation to lean/understand it ? 

The compiler does the right thing, just verifier is not advanced enough.
Is it worthy to share this issue of verifier.c with bpf maintainers ? The compiler which is used here is clang which is called by bcc, right ?

Yes, you will need some source workaround. You could try below (untested):
+ udp_len = udp_len & 0x1ff;

I tested it and it seems to work. Thx a lot !!

But that means I can not use the u16 max value ?


Re: math between pkt pointer and register with unbounded min value is not allowed #verifier

Jiong Wang
 

Yonghong Song writes:

On Wed, Mar 6, 2019 at 7:08 AM <contact@...> wrote:

I'm playing with bcc to prototype an UDP load balancer.

I'm facing an issue that I didn't succeed to understand...

In my code I tried to validate my UDP packet using code like this :

struct udphdr *udp;
udp = iph + 1;
if (udp + 1 > data_end)
return XDP_DROP;
__u16 udp_len = bpf_ntohs(udp->len);
//__u16 udp_len = 8;
if (udp_len < 8)
return XDP_DROP;
if (udp_len > 512) // TODO use a more approriate max value
return XDP_DROP;
if ((void *) udp + udp_len > data_end)
return XDP_DROP;

And the verifier does not like it ..
This is caused by compiler optimizations.


28: (71) r2 = *(u8 *)(r7 +23)
29: (b7) r0 = 2
30: (55) if r2 != 0x11 goto pc+334
R0=inv2 R1=pkt_end(id=0,off=0,imm=0) R2=inv17 R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=34,imm=0) R8=pkt(id=0,off=34,r=34,imm=0) R9=pkt(id=0,off=14,r=34,imm=0) R10=fp0,call_-1
31: (bf) r2 = r8
32: (07) r2 += 8
33: (b7) r0 = 1
34: (2d) if r2 > r1 goto pc+330
R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=pkt(id=0,off=42,r=42,imm=0) R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1
35: (69) r3 = *(u16 *)(r7 +38)
36: (dc) r3 = be16 r3
r3 get the value from memory, its value could be any one as permitted
by the type.

37: (bf) r2 = r3
38: (07) r2 += -8
39: (57) r2 &= 65535
40: (b7) r0 = 1
41: (25) if r2 > 0x1f8 goto pc+323
test is done by r2. We indeed get better range for r2 (below:
R2=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) )
but r3 range is not tightened.
I had run into similar issue when debugging some other rejection before
JMP32 introduced when LLVM was generating similar sequences under defult
64-bit mode, but IIRC LLVM generates betweer sequences with -mattr=alu32,
under which it will just use w3 (as the type should be optimized into
32-bit) for the comparison.

So, I guess this testcase could have easier sequence for verifier under
ALU32 mode. But for this case, BPF_END is used which doesn't have
sub-register code-gen support inside LLVM for be16 and be32 at the moment
(noticed this several days ago when doing some other benchmarking).

If I have .i file, I could do a quick prototype to see if ALU32 could
improve this.

Regards,
Jiong


Re: math between pkt pointer and register with unbounded min value is not allowed #verifier

Yonghong Song
 

On Wed, Mar 6, 2019 at 7:08 AM <contact@...> wrote:

I'm playing with bcc to prototype an UDP load balancer.

I'm facing an issue that I didn't succeed to understand...

In my code I tried to validate my UDP packet using code like this :

struct udphdr *udp;
udp = iph + 1;
if (udp + 1 > data_end)
return XDP_DROP;
__u16 udp_len = bpf_ntohs(udp->len);
//__u16 udp_len = 8;
if (udp_len < 8)
return XDP_DROP;
if (udp_len > 512) // TODO use a more approriate max value
return XDP_DROP;
if ((void *) udp + udp_len > data_end)
return XDP_DROP;

And the verifier does not like it ..
This is caused by compiler optimizations.


28: (71) r2 = *(u8 *)(r7 +23)
29: (b7) r0 = 2
30: (55) if r2 != 0x11 goto pc+334
R0=inv2 R1=pkt_end(id=0,off=0,imm=0) R2=inv17 R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=34,imm=0) R8=pkt(id=0,off=34,r=34,imm=0) R9=pkt(id=0,off=14,r=34,imm=0) R10=fp0,call_-1
31: (bf) r2 = r8
32: (07) r2 += 8
33: (b7) r0 = 1
34: (2d) if r2 > r1 goto pc+330
R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=pkt(id=0,off=42,r=42,imm=0) R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1
35: (69) r3 = *(u16 *)(r7 +38)
36: (dc) r3 = be16 r3
r3 get the value from memory, its value could be any one as permitted
by the type.

37: (bf) r2 = r3
38: (07) r2 += -8
39: (57) r2 &= 65535
40: (b7) r0 = 1
41: (25) if r2 > 0x1f8 goto pc+323
test is done by r2. We indeed get better range for r2 (below:
R2=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) )
but r3 range is not tightened.

R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) R3=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1
42: (bf) r2 = r7
43: (0f) r2 += r3
math between pkt pointer and register with unbounded min value is not allowed
Here, we use r3 to do arith and the verifier not happy. If we use the
tightened old r2, it may work.
The compiler does the right thing, just verifier is not advanced enough.


I'm pretty sure the issue is about udp_len, that's why I tried to validate its value before to use it ... but without success...
When I set udp_len to 8 (just for testing) this seems to works. Any idea about that ?
Yes, you will need some source workaround. You could try below (untested):

if (udp_len > 512) // TODO use a more approriate max value
return XDP_DROP;
+ udp_len = udp_len & 0x1ff;
if ((void *) udp + udp_len > data_end)
return XDP_DROP;


Full code is available here : https://gist.github.com/sbernard31/d4fee7518a1ff130452211c0d355b3f7

(I'm using python-bpfcc 0.8.0-4 from debian sid with a 4.19.12 kernel)
(I don't know if this is the right place for this kind of question, )


Re: Sharing eBPF map between two eBPF kernel programs

Yonghong Song
 

On Mon, Mar 4, 2019 at 10:45 PM Kanthi P <pavuluri.kanthi@...> wrote:

Thanks for that!

I think those work because bcc has given utilities to do that.

I am using C and we use libbpf to load programs and to do map operations.

Do you know if we can achieve it in this way too?
You should be able to do the work with C as well. Just use APIs from libbpf.h,
e.g., bpf_prog_get_next_id, bpf_prog_get_fd_by_id, bpf_map_get_fd_by_id.
Yes, we may miss bpf_map_get_next_id and bpf_obj_get_info_by_fd.
You can just add prototype to bcc/src/cc/libbpf.h and the implementation
is libbpf submodule.

We can add the missing piece in bcc/src/cc/{libbpf.h, libbpf.c}.


Thanks,
Kanthi

On Tue, Mar 5, 2019 at 2:21 AM Y Song <ys114321@...> wrote:

On Mon, Mar 4, 2019 at 9:52 AM Kanthi P <Pavuluri.kanthi@...> wrote:

Hi,

I have two eBPF programs, each with its own user and kernel programs.

One of the programs defines an eBPF map.

How do I access this map from another eBPF kernel program?
In examples/cpp, we have UseExternalMap.cc, which gives an example of
how to share maps between two processes with C++ interface.
Basically, if you know the ID of the map in another application (you
can see id use bpftool), you can get a fd for that map by ID with API
`bpf_map_get_fd_by_id()`, and example shows how to inject the new `fd`
to bcc compilation system.

We do not have a python way to share map yet bwtween two different processes.

Please take a look. If you feel the interface needs to improve to
support your use case, we can do that.


PS:

I can pin the map and that helps me access this map from other program's user space.
But looking for a direct way in which I can access this map from other programs's kernel space.

Thanks!


minutes: IO Visor TSC/Dev Meeting

Brenden Blanco
 

Hi All,

Thank you for joining the call today. Here are my notes from the discussion.

Thanks,
Brenden

=== Discussion ===
Brenden:
* Plan to tag release to coincide with kernel 5.0

Brendan:
* Speaking this weekend at SCaLE in Los Angeles

Yonghong:
* LLVM work
* compile once - run anywhere WIP
* support for static variables

Daniel:
* Global data support work in kernel continues
* Ability to lock maps as read-only
* bugfixes after merge window

Alexei:
* Some thoughts on future work of BPF
* especially with introduction of BTF
* overall needs concerted effort to improve debuggability
* BTF for programs itself with source/type/layout information
* structures for maps and global data
* suggest to always require type information
(already turned on by default in bcc and supported by llvm)
* Some extra hoops to jump through for driver embedded BPF
* to be enabled with a sysctl
* kernel support is ready
* some long tail of support - e.g. systemd has raw assembly BPF
* kconfig option - eventual deprecation
* if kernel is default strict, llvm should automatically emit BTF as well
* memcg accounting patch status?
* Daniel - still being worked on
* proposal to enable the same accounting for verifier memory
* helps to enable verifier multithreading

Jakub:
* question regarding global data atomicity
* Daniel - requires read once / write once instructions to work properly
* some todo work on documentation, interpreter + jit implementations
* depends on architecture (machine word size guarantees only)

Jesper:
* which llvm release supports BTF
* landed in December - will be in 8.0, better in 9.0
* working on tutorial for xdp at netdev
* https://www.netdevconf.org/0x13/session.html?tutorial-XDP-hands-on
* soliciting feedback
* https://github.com/xdp-project/xdp-tutorial/

Saeed:
* request to devote some time in the next meeting to iron out some XDP issues
* please send an agend in reply to the reminder email before next call
* prepare discussion over email in between time

=== Attendees ===
Alexei Starovoitov
Marco Leogrande
Mauricio Vasquez
Paul Chaignon
Brenden Blanco
Jiong Wang
Yonghong Song
Daniel Borkmann
Jesper Brouer
Quentin Monnet
Dan Siemon
Jakub Kicinski
Saeed
John
Yutaro


math between pkt pointer and register with unbounded min value is not allowed #verifier

Simon
 

I'm playing with bcc to prototype an UDP load balancer.

I'm facing an issue that I didn't succeed to understand...

In my code I tried to validate my UDP packet using code like this :

    struct udphdr *udp;
    udp = iph + 1;
    if (udp + 1 > data_end)
        return XDP_DROP;
    __u16 udp_len = bpf_ntohs(udp->len);
    //__u16 udp_len = 8;
    if (udp_len < 8)
        return XDP_DROP;
    if (udp_len > 512) // TODO use a more approriate max value
        return XDP_DROP;
    if ((void *) udp + udp_len > data_end)
        return XDP_DROP;
And the verifier does not like it ..
28: (71) r2 = *(u8 *)(r7 +23)
29: (b7) r0 = 2
30: (55) if r2 != 0x11 goto pc+334
 R0=inv2 R1=pkt_end(id=0,off=0,imm=0) R2=inv17 R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=34,imm=0) R8=pkt(id=0,off=34,r=34,imm=0) R9=pkt(id=0,off=14,r=34,imm=0) R10=fp0,call_-1
31: (bf) r2 = r8
32: (07) r2 += 8
33: (b7) r0 = 1
34: (2d) if r2 > r1 goto pc+330
 R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=pkt(id=0,off=42,r=42,imm=0) R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1
35: (69) r3 = *(u16 *)(r7 +38)
36: (dc) r3 = be16 r3
37: (bf) r2 = r3
38: (07) r2 += -8
39: (57) r2 &= 65535
40: (b7) r0 = 1
41: (25) if r2 > 0x1f8 goto pc+323
 R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) R3=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1
42: (bf) r2 = r7
43: (0f) r2 += r3
math between pkt pointer and register with unbounded min value is not allowed

I'm pretty sure the issue is about udp_len, that's why I tried to validate its value before to use it ... but without success...
When I set udp_len to 8 (just for testing) this seems to works. Any idea about that ?

Full code is available here : https://gist.github.com/sbernard31/d4fee7518a1ff130452211c0d355b3f7

(I'm using python-bpfcc  0.8.0-4 from debian sid with a 4.19.12 kernel)
(I don't know if this is the right place for this kind of question, )


reminder: IO Visor TSC/Dev Meeting

Brenden Blanco
 

Please join us tomorrow for our bi-weekly call. As usual, this meeting is
open to everybody and completely optional.
You might be interested to join if:
You want to know what is going on in BPF land
You are doing something interesting yourself with BPF and would like to share
You want to know what the heck BPF is

=== IO Visor Dev/TSC Meeting ===

Every 2 weeks on Wednesday, from Wednesday, January 25, 2017, to no end date
11:00 am | Pacific Daylight Time (San Francisco, GMT-07:00) | 30 min

https://bluejeans.com/568677804/

https://www.timeanddate.com/worldclock/meetingdetails.html?year=2019&month=3&day=6&hour=19&min=0&sec=0&p1=900


Re: Sharing eBPF map between two eBPF kernel programs

Kanthi P
 

Thanks for that!

I think those work because bcc has given utilities to do that.

I am using C and we use libbpf to load programs and to do map operations. 

Do you know if we can achieve it in this way too?

Thanks,
Kanthi

On Tue, Mar 5, 2019 at 2:21 AM Y Song <ys114321@...> wrote:
On Mon, Mar 4, 2019 at 9:52 AM Kanthi P <Pavuluri.kanthi@...> wrote:
>
> Hi,
>
> I have two eBPF programs, each with its own user and kernel programs.
>
> One of the programs defines an eBPF map.
>
> How do I access this map from another eBPF kernel program?

In examples/cpp, we have UseExternalMap.cc, which gives an example of
how to share maps between two processes with C++ interface.
Basically, if you know the ID of the map in another application (you
can see id use bpftool), you can get a fd for that map by ID with API
`bpf_map_get_fd_by_id()`, and example shows how to inject the new `fd`
to bcc compilation system.

We do not have a python way to share map yet bwtween two different processes.

Please take a look. If you feel the interface needs to improve to
support your use case, we can do that.

>
> PS:
>
> I can pin the map and that helps me access this map from other program's user space.
> But looking for a direct way in which I can access this map from other programs's kernel space.
>
> Thanks!


Re: Sharing eBPF map between two eBPF kernel programs

Yonghong Song
 

On Mon, Mar 4, 2019 at 9:52 AM Kanthi P <Pavuluri.kanthi@...> wrote:

Hi,

I have two eBPF programs, each with its own user and kernel programs.

One of the programs defines an eBPF map.

How do I access this map from another eBPF kernel program?
In examples/cpp, we have UseExternalMap.cc, which gives an example of
how to share maps between two processes with C++ interface.
Basically, if you know the ID of the map in another application (you
can see id use bpftool), you can get a fd for that map by ID with API
`bpf_map_get_fd_by_id()`, and example shows how to inject the new `fd`
to bcc compilation system.

We do not have a python way to share map yet bwtween two different processes.

Please take a look. If you feel the interface needs to improve to
support your use case, we can do that.


PS:

I can pin the map and that helps me access this map from other program's user space.
But looking for a direct way in which I can access this map from other programs's kernel space.

Thanks!


Sharing eBPF map between two eBPF kernel programs

Kanthi P
 

Hi,

I have two eBPF programs, each with its own user and kernel programs.

One of the programs defines an eBPF map.

How do I access this map from another eBPF kernel program?

PS:

I can pin the map and that helps me access this map from other program's user space.
But looking for a direct way in which I can access this map from other programs's kernel space.

Thanks!


about kprobe/kretprobe matching

ovinee@...
 

Hi,

I'm trying to access the packet content in tcp_transmit_skb using eBPF/BCC.
I defined kprobe____tcp_transmit_skb/kretprobe____tcp_transmit_skb functions.
Since in kretprobe we cannot access the function arguments (I'm not sure), I stored skb(struct sk_buff *) in kprobe____tcp_transmit_skb by using the pid as the key.

u64 pid = bpf_get_current_pid_tgid();
curr_skb.update(&pid, &skb);

However, the issue is that tcp_transmit_skb function can be called multiple times before kretprobe____tcp_transmit_skb is executed with the same pid.
So I cannot correctly match between kprobe____tcp_transmit_skb and kretprobe____tcp_transmit_skb functions for the same skb.
Is there any good way to match between kprobe and kretprobe for the same function call?
Or is there any method that I can access the function arguments in kretprobe?

Thanks,
Youngbin


share maps for two prog

Иван Иванов
 

I have two port running xdp prog. Each have same type, struct and purpose maps. I want to use shared map. So tested such a way, run xdp prog on each device with only one func - bpf_tail_call. This calls to the third xdp prog and gets and saves data to its maps. So is it only such a way, or may be another way? 
If only one, how can i control, that bpf_tail_call not drop some packets?
And what "cost" have bpf_tail_call? Is it cheap or not? Or its better to send to main(third) prog to save data in man not xdp_md , but may by some (8 bytes) needed to save data?


Re: XDP Offloading

Andrew Gospodarek
 

On Mon, Feb 25, 2019 at 08:50:52AM -0800, mdimolianis@... wrote:
Hi all,
I am wondering except for Netronome Cards if there are any other NICs that support XDP Offloading?
Thank you in advance!
Many of the ARM-based SmartNICs[1] allow users to download and execute
code on the cards themselves to save the server cores from being used
for excuting the XDP programs.

The Broadcom Stingray/PS225[2] is one of those cards.

This is possible as the NIC cards themselves are also running full Linux
distros with console access so you can treat it as a server inside your
server.

1. I define a SmartNIC NIC where control and dataplane traffic can be
offload from server cores to a NIC with general purpose ARM cores
running Linux.

2. https://www.broadcom.com/products/ethernet-connectivity/smartnic/ps225


Re: XDP Offloading

Raymond
 

On 2019-02-25 9:50 a.m., mdimolianis@... wrote:
I am wondering except for Netronome Cards if there are any other NICs that support XDP Offloading?
Take a look at https://github.com/iovisor/bcc/blob/master/docs/kernel-versions.md

It shows the state of various drivers.  I don't have links handy, but some drivers offer driver based off-load, and some will push to the hardware for offload.

Mellanox might do hardware offload.  I thought I read that some 10/40G Intel cards can do it, or maybe something new coming down the pipe.  Can't remember at the moment.

Take a look at https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/ which has the word offload in there.  Most references might be Netronome related.  But one mentioned Agilio, which may or may not be applicable.

Raymond
https://blog.raymond.burkholder.net


Re: XDP Offloading

Иван Иванов
 


I think not now, because i asked the same question 2-3 months ago i answer was the same.
--
Отправлено из мобильного приложения Яндекс.Почты


XDP Offloading

mdimolianis@...
 

Hi all,
I am wondering except for Netronome Cards if there are any other NICs that support XDP Offloading?
Thank you in advance!


Re: kretprobe not consistently being triggered for sys_futex calls?

Paul Chaignon
 

On Tue, Feb 12, 2019 at 07:02:58AM -0800, Ashley wrote:
So I upgraded my kernel version to 5.0.0-rc6, and it seems to consistently
work perfectly now, even with a much higher density of futex calls than I
was working with before.

I'm still curious what the issue is, and I sadly can't say with any
confidence that it's definitely the kernel version, since I accidentally
broke my filesystem trying to resize it and had to make a fresh VM, so I
may have different BCC or dependency versions installed on this machine,
but I'm hoping to find time to dig a bit more in a few weeks to see if I
can determine what was going on here.
bcc now supports the maxactive kretprobe parameter [1], so if you want to
test whether a higher maxactive value solves the issue you had, it's as
easy as calling BPF.attach_kretprobe with a maxactive=your_value argument.

1 - https://github.com/iovisor/bcc/pull/2224


Thanks again for the help, Paul! I wouldn't have considered playing with
the kernel version without realising it was working fine  for the kernel
module.

Regards,
Ashley


Re: add prog id to prog map

Иван Иванов
 

map updates, but next xdp prog do not work, so i think i make something wrong or does not understand something

7:48, 21 февраля 2019 г., Jakub Kicinski <jakub.kicinski@...>:

On Thu, 21 Feb 2019 07:33:57 +0300, Иван Иванов wrote:

 I loaded tail call prog with map via ip util, i do not know, if it
 pin map. But i can see it in list of maps via bpftool map with id


Try:

bpftool map pin id $YOUR_ID /sys/fs/bpf/my_map

and see if the prog array updates stick after that.



--
Отправлено из мобильного приложения Яндекс.Почты


Re: add prog id to prog map

Jakub Kicinski
 

On Thu, 21 Feb 2019 07:33:57 +0300, Иван Иванов wrote:
I loaded tail call prog with map via ip util, i do not know, if it
pin map. But i can see it in list of maps via bpftool map with id
Try:

bpftool map pin id $YOUR_ID /sys/fs/bpf/my_map

and see if the prog array updates stick after that.

421 - 440 of 2021