|
bpf: Failed to load program: Permission denied
Jacob Steadman
Hi,
I'm new to BPF. I'm trying to write a program that analyses the structure o= f DNS requests. I keep getting the following error (bellow) at a certain point in the code (bellow). The error only occurs when I try to "return -1;" (i.e. allow the packet). I= f I remove this line the program executes as expected. I wonder if it could be an issue with the kernel version rather than the co= de? (Ubuntu 16.04.4 LTS, Kernel version 4.4.0-87-generic) Error*************************************** bpf: Failed to load program: Permission denied ... ... R2 invalid mem access 'inv' HINT: The invalid mem access 'inv' error can happen if you try to dereferen= ce memory without first using bpf_probe_read() to copy it to the BPF stack.= Sometimes the bpf_probe_read is automatic by the bcc rewriter, other times= you'll need to be explicit. Traceback (most recent call last): File "dns_matching.py", line 57, in <module> function_dns_matching =3D bpf.load_func("dns_exfil_detection_v2", BPF.S= OCKET_FILTER) File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 379, in loa= d_func (func_name, errstr)) Exception: Failed to load BPF program dns_exfil_detection_v2: Permission de= nied ******************************************** Code**************************************** #pragma unroll for(i =3D 0; i<255;i++){ c =3D cursor_advance(cursor, 1); if (c->c =3D=3D 0) break; key.p[i] =3D c->c; //**ensure this is the correct max length of a subdomain** if(c->c < 0x0f){ subdomLengths[subdomainCount] =3D (u16) c->= c; subdomainCount =3D subdomainCount +1; } } *** if(subdomLengths[subdomainCount] =3D=3D 2 && subdomLengths[subdomainCou= nt-1] =3D=3D2 && subdomainCount <4 ){ *** return -1; *** } *********************************************
|
|
|
|
Re: R? min value is negative, either use unsigned or 'var &= const'
#verifier
Simon
Here a python 2 version.
|
|
|
|
Re: R? min value is negative, either use unsigned or 'var &= const'
#verifier
Yonghong Song
On Mon, Mar 11, 2019 at 4:08 AM Simon <contact@...> wrote:
I took a brief look. Indeed, it is very strange. I can see proper value of udp_len is stored into r10 - 40, but when it is retrieved later, the value became unkown.... I will try to experiment with this problem later this week. Could you do me a favor to make it reproducible with python2? My env. flexible to rebuild/retry with kernel is python2 friendly.
|
|
|
|
Re: R? min value is negative, either use unsigned or 'var &= const'
#verifier
Simon
I tried to understand again this verifier error again and probably my previous post does not contain enough information. I understand that : 93: (67) r0 <<= 32 294: (c7) r0 s>>= 32 295: (b7) r1 = 0 296: (b7) r2 = 0 297: (bf) r3 = r8 298: (79) r4 = *(u64 *)(r10 -40) 299: (bf) r5 = r0 300: (85) call bpf_csum_diff#28 R4 min value is negative, either use unsigned or 'var &= const' is about this line (in *csum = bpf_csum_diff(0, 0, data_start, data_size, *csum); R1=0, 298: (79) r4 = *(u64 *)(r10 -40) As I understand this line, r4 will get a value in the stack ( (By the way I can not understand why this is a u64 and not a u16 as I tried to use the &= tricks like : data_size = data_size & 0x1ff; Same issue ... R0=inv(id=0,umax_value=4295032831,var_off=(0x0; 0x1ffffffff)) I reference the commit instead of repository to keep the link consistent over the time : https://github.com/sbernard31/udploadbalancer/tree/5ca93d0893a60bc70a75f30eb5cfde496a9e5d93 Again do not hesitate to redirect me to better place if I'm not asking at the right place :) Thx again for your time.
|
|
|
|
Re: math between pkt pointer and register with unbounded min value is not allowed
#verifier
Yonghong Song
On Fri, Mar 8, 2019 at 9:22 AM Simon <contact@...> wrote:
The be16 is to convert r3 with big endian encoding. If the host system is big endian, it will do nothing. Otherwise, it will convert from little endian to big endian. Yes, there is no documentation. It intends to be self explanatory. I guess "be16" is special and may need some documentation. Otherwise assembly-style codes should be easy to understand. I am also a regular kernel/bpf reviewer. The bpf maintainers/community are aware of this limitation. As you mentioned, the verifier is already very complex. To implement complex tracking like described in this thread will make verifier even more complex, hence this is delayed. One of reason is that we have reasonable, although unpleasant, workarounds. Yes, it is compiled with clang. You can. I add that because you have a test to limit the range of the value to 511.
|
|
|
|
Re: minutes: IO Visor TSC/Dev Meeting
Saeed Mahameed
Hi Guys,
toggle quoted messageShow quoted text
My agenda for next meeting: 1) unifying and centralizing XDP statistics accounting [1]. 2) XDP resource management, User API [2]. 3) XDP meta data via btf (in kernel BTF registration). 4) all of the above issues share one common problem, which is the lack of a unified user interface without it, We really can't make a real progress. I just sent a proposal [3] for away to achieve the unified interface, please look it up and let me know your thoughts. [1] https://github.com/xdp-project/xdp-project/blob/master/xdp-project.org#statistics-per-xdp-action [2] https://github.com/xdp-project/xdp-project/blob/master/xdp-project.org#better-ndo_xdp_xmit-resource-management [3] Subject: "[RFC][Proposal] BPF Control MAP" Thanks, Saeed
On Wed, Mar 6, 2019 at 12:51 PM Brenden Blanco <bblanco@...> wrote:
|
|
|
|
R? min value is negative, either use unsigned or 'var &= const'
#verifier
Simon
I suspect I fall in another issue with verifier.
This code (taking from katran : I just changed data_size type to __u32): __attribute__((__always_inline__))
static inline void ipv4_l4_csum(void *data_start, __u32 data_size,
__u64 *csum, struct iphdr *iph) {
__u32 tmp = 0;
*csum = bpf_csum_diff(0, 0, &iph->saddr, sizeof(__be32), *csum);
*csum = bpf_csum_diff(0, 0, &iph->daddr, sizeof(__be32), *csum);
tmp = __builtin_bswap32((__u32)(iph->protocol));
*csum = bpf_csum_diff(0, 0, &tmp, sizeof(__u32), *csum);
tmp = __builtin_bswap32((__u32)(data_size));
*csum = bpf_csum_diff(0, 0, &tmp, sizeof(__u32), *csum);
*csum = bpf_csum_diff(0, 0, data_start, data_size, *csum);
*csum = csum_fold_helper(*csum);
}
But it brings to this verifier issue :
93: (67) r0 <<= 32 294: (c7) r0 s>>= 32 295: (b7) r1 = 0 296: (b7) r2 = 0 297: (bf) r3 = r8 298: (79) r4 = *(u64 *)(r10 -40) 299: (bf) r5 = r0 300: (85) call bpf_csum_diff#28 R4 min value is negative, either use unsigned or 'var &= const'The whole code is available here. This is maybe relative to my previous issue.
|
|
|
|
Re: math between pkt pointer and register with unbounded min value is not allowed
#verifier
Simon
35: (69) r3 = *(u16 *)(r7 +38)r3 get the value from memory, its value could be any one as permitted Does it mean that r3 is considered as be16 ? I do not understand why as I explicitly convert it in u16. This output language is a readable format of bpf bytecode, right ? Is there any documentation to lean/understand it ? The compiler does the right thing, just verifier is not advanced enough.Is it worthy to share this issue of verifier.c with bpf maintainers ? The compiler which is used here is clang which is called by bcc, right ? Yes, you will need some source workaround. You could try below (untested): I tested it and it seems to work. Thx a lot !! But that means I can not use the u16 max value ?
|
|
|
|
Re: math between pkt pointer and register with unbounded min value is not allowed
#verifier
Jiong Wang
Yonghong Song writes:
On Wed, Mar 6, 2019 at 7:08 AM <contact@...> wrote:I had run into similar issue when debugging some other rejection beforeThis is caused by compiler optimizations. JMP32 introduced when LLVM was generating similar sequences under defult 64-bit mode, but IIRC LLVM generates betweer sequences with -mattr=alu32, under which it will just use w3 (as the type should be optimized into 32-bit) for the comparison. So, I guess this testcase could have easier sequence for verifier under ALU32 mode. But for this case, BPF_END is used which doesn't have sub-register code-gen support inside LLVM for be16 and be32 at the moment (noticed this several days ago when doing some other benchmarking). If I have .i file, I could do a quick prototype to see if ALU32 could improve this. Regards, Jiong
|
|
|
|
Re: math between pkt pointer and register with unbounded min value is not allowed
#verifier
Yonghong Song
On Wed, Mar 6, 2019 at 7:08 AM <contact@...> wrote:
This is caused by compiler optimizations. r3 get the value from memory, its value could be any one as permitted by the type. 37: (bf) r2 = r3test is done by r2. We indeed get better range for r2 (below: R2=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) ) but r3 range is not tightened. R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) R3=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1Here, we use r3 to do arith and the verifier not happy. If we use the tightened old r2, it may work. The compiler does the right thing, just verifier is not advanced enough. Yes, you will need some source workaround. You could try below (untested): if (udp_len > 512) // TODO use a more approriate max value return XDP_DROP; + udp_len = udp_len & 0x1ff; if ((void *) udp + udp_len > data_end) return XDP_DROP;
|
|
|
|
Re: Sharing eBPF map between two eBPF kernel programs
Yonghong Song
On Mon, Mar 4, 2019 at 10:45 PM Kanthi P <pavuluri.kanthi@...> wrote:
You should be able to do the work with C as well. Just use APIs from libbpf.h, e.g., bpf_prog_get_next_id, bpf_prog_get_fd_by_id, bpf_map_get_fd_by_id. Yes, we may miss bpf_map_get_next_id and bpf_obj_get_info_by_fd. You can just add prototype to bcc/src/cc/libbpf.h and the implementation is libbpf submodule. We can add the missing piece in bcc/src/cc/{libbpf.h, libbpf.c}.
|
|
|
|
minutes: IO Visor TSC/Dev Meeting
Brenden Blanco
Hi All,
Thank you for joining the call today. Here are my notes from the discussion. Thanks, Brenden === Discussion === Brenden: * Plan to tag release to coincide with kernel 5.0 Brendan: * Speaking this weekend at SCaLE in Los Angeles Yonghong: * LLVM work * compile once - run anywhere WIP * support for static variables Daniel: * Global data support work in kernel continues * Ability to lock maps as read-only * bugfixes after merge window Alexei: * Some thoughts on future work of BPF * especially with introduction of BTF * overall needs concerted effort to improve debuggability * BTF for programs itself with source/type/layout information * structures for maps and global data * suggest to always require type information (already turned on by default in bcc and supported by llvm) * Some extra hoops to jump through for driver embedded BPF * to be enabled with a sysctl * kernel support is ready * some long tail of support - e.g. systemd has raw assembly BPF * kconfig option - eventual deprecation * if kernel is default strict, llvm should automatically emit BTF as well * memcg accounting patch status? * Daniel - still being worked on * proposal to enable the same accounting for verifier memory * helps to enable verifier multithreading Jakub: * question regarding global data atomicity * Daniel - requires read once / write once instructions to work properly * some todo work on documentation, interpreter + jit implementations * depends on architecture (machine word size guarantees only) Jesper: * which llvm release supports BTF * landed in December - will be in 8.0, better in 9.0 * working on tutorial for xdp at netdev * https://www.netdevconf.org/0x13/session.html?tutorial-XDP-hands-on * soliciting feedback * https://github.com/xdp-project/xdp-tutorial/ Saeed: * request to devote some time in the next meeting to iron out some XDP issues * please send an agend in reply to the reminder email before next call * prepare discussion over email in between time === Attendees === Alexei Starovoitov Marco Leogrande Mauricio Vasquez Paul Chaignon Brenden Blanco Jiong Wang Yonghong Song Daniel Borkmann Jesper Brouer Quentin Monnet Dan Siemon Jakub Kicinski Saeed John Yutaro
|
|
|
|
math between pkt pointer and register with unbounded min value is not allowed
#verifier
Simon
I'm playing with bcc to prototype an UDP load balancer. I'm facing an issue that I didn't succeed to understand... In my code I tried to validate my UDP packet using code like this : struct udphdr *udp;
udp = iph + 1;
if (udp + 1 > data_end)
return XDP_DROP;
__u16 udp_len = bpf_ntohs(udp->len);
//__u16 udp_len = 8;
if (udp_len < 8)
return XDP_DROP;
if (udp_len > 512) // TODO use a more approriate max value
return XDP_DROP;
if ((void *) udp + udp_len > data_end)
return XDP_DROP;
And the verifier does not like it ..
28: (71) r2 = *(u8 *)(r7 +23) 29: (b7) r0 = 2 30: (55) if r2 != 0x11 goto pc+334 R0=inv2 R1=pkt_end(id=0,off=0,imm=0) R2=inv17 R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=34,imm=0) R8=pkt(id=0,off=34,r=34,imm=0) R9=pkt(id=0,off=14,r=34,imm=0) R10=fp0,call_-1 31: (bf) r2 = r8 32: (07) r2 += 8 33: (b7) r0 = 1 34: (2d) if r2 > r1 goto pc+330 R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=pkt(id=0,off=42,r=42,imm=0) R3=inv5 R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1 35: (69) r3 = *(u16 *)(r7 +38) 36: (dc) r3 = be16 r3 37: (bf) r2 = r3 38: (07) r2 += -8 39: (57) r2 &= 65535 40: (b7) r0 = 1 41: (25) if r2 > 0x1f8 goto pc+323 R0=inv1 R1=pkt_end(id=0,off=0,imm=0) R2=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) R3=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=pkt(id=0,off=0,r=42,imm=0) R8=pkt(id=0,off=34,r=42,imm=0) R9=pkt(id=0,off=14,r=42,imm=0) R10=fp0,call_-1 42: (bf) r2 = r7 43: (0f) r2 += r3 math between pkt pointer and register with unbounded min value is not allowed I'm pretty sure the issue is about (I'm using python-bpfcc 0.8.0-4 from debian sid with a 4.19.12 kernel)
|
|
|
|
reminder: IO Visor TSC/Dev Meeting
Brenden Blanco
Please join us tomorrow for our bi-weekly call. As usual, this meeting is
open to everybody and completely optional. You might be interested to join if: You want to know what is going on in BPF land You are doing something interesting yourself with BPF and would like to share You want to know what the heck BPF is === IO Visor Dev/TSC Meeting === Every 2 weeks on Wednesday, from Wednesday, January 25, 2017, to no end date 11:00 am | Pacific Daylight Time (San Francisco, GMT-07:00) | 30 min https://bluejeans.com/568677804/ https://www.timeanddate.com/worldclock/meetingdetails.html?year=2019&month=3&day=6&hour=19&min=0&sec=0&p1=900
|
|
|
|
Re: Sharing eBPF map between two eBPF kernel programs
Kanthi P <Pavuluri.kanthi@...>
Thanks for that! I think those work because bcc has given utilities to do that. I am using C and we use libbpf to load programs and to do map operations. Do you know if we can achieve it in this way too? Thanks, Kanthi
On Tue, Mar 5, 2019 at 2:21 AM Y Song <ys114321@...> wrote: On Mon, Mar 4, 2019 at 9:52 AM Kanthi P <Pavuluri.kanthi@...> wrote:
|
|
|
|
Re: Sharing eBPF map between two eBPF kernel programs
Yonghong Song
On Mon, Mar 4, 2019 at 9:52 AM Kanthi P <Pavuluri.kanthi@...> wrote:
In examples/cpp, we have UseExternalMap.cc, which gives an example of how to share maps between two processes with C++ interface. Basically, if you know the ID of the map in another application (you can see id use bpftool), you can get a fd for that map by ID with API `bpf_map_get_fd_by_id()`, and example shows how to inject the new `fd` to bcc compilation system. We do not have a python way to share map yet bwtween two different processes. Please take a look. If you feel the interface needs to improve to support your use case, we can do that.
|
|
|
|
Sharing eBPF map between two eBPF kernel programs
Kanthi P <Pavuluri.kanthi@...>
Hi, I have two eBPF programs, each with its own user and kernel programs. One of the programs defines an eBPF map. How do I access this map from another eBPF kernel program? PS: I can pin the map and that helps me access this map from other program's user space. But looking for a direct way in which I can access this map from other programs's kernel space. Thanks!
|
|
|
|
about kprobe/kretprobe matching
ovinee@...
Hi,
I'm trying to access the packet content in tcp_transmit_skb using eBPF/BCC. I defined kprobe____tcp_transmit_skb/kretprobe____tcp_transmit_skb functions. Since in kretprobe we cannot access the function arguments (I'm not sure), I stored skb(struct sk_buff *) in kprobe____tcp_transmit_skb by using the pid as the key. u64 pid = bpf_get_current_pid_tgid(); curr_skb.update(&pid, &skb); However, the issue is that tcp_transmit_skb function can be called multiple times before kretprobe____tcp_transmit_skb is executed with the same pid. So I cannot correctly match between kprobe____tcp_transmit_skb and kretprobe____tcp_transmit_skb functions for the same skb. Is there any good way to match between kprobe and kretprobe for the same function call? Or is there any method that I can access the function arguments in kretprobe? Thanks, Youngbin
|
|
|
|
share maps for two prog
Иван Иванов
I have two port running xdp prog. Each have same type, struct and purpose maps. I want to use shared map. So tested such a way, run xdp prog on each device with only one func - bpf_tail_call. This calls to the third xdp prog and gets and saves data to its maps. So is it only such a way, or may be another way?
If only one, how can i control, that bpf_tail_call not drop some packets? And what "cost" have bpf_tail_call? Is it cheap or not? Or its better to send to main(third) prog to save data in man not xdp_md , but may by some (8 bytes) needed to save data?
|
|
|
|
Re: XDP Offloading
Andrew Gospodarek
On Mon, Feb 25, 2019 at 08:50:52AM -0800, mdimolianis@... wrote:
Hi all,Many of the ARM-based SmartNICs[1] allow users to download and execute code on the cards themselves to save the server cores from being used for excuting the XDP programs. The Broadcom Stingray/PS225[2] is one of those cards. This is possible as the NIC cards themselves are also running full Linux distros with console access so you can treat it as a server inside your server. 1. I define a SmartNIC NIC where control and dataplane traffic can be offload from server cores to a NIC with general purpose ARM cores running Linux. 2. https://www.broadcom.com/products/ethernet-connectivity/smartnic/ps225
|
|
|