The other day I was in the process of porting a little libbpf application from Ubuntu 20 (Linux 5.4) to CentOS 8 (Linux 4.18). This program uses tracepoint:tcp:tcp_send_reset. Here's the relevant BPF code:
struct tcp_send_rst_args {
long long pad;
const void * skbaddr;
const void * skaddr;
int state;
u16 sport;
u16 dport;
u8 saddr[4];
u8 daddr[4];
u8 saddr_v6[16];
u8 daddr_v6[16];
};
SEC("tracepoint/tcp/tcp_send_reset")
int tcp_send_reset_prog(struct tcp_send_rst_args * args) {
struct tcprstsend_data_t data = {};
data.pid = bpf_get_current_pid_tgid() >> 32;
data.sport = args->sport;
data.dport = args->dport;
bpf_get_current_comm(&data.comm, sizeof(data.comm));
__builtin_memcpy(&data.saddr, args->saddr, sizeof(data.saddr));
__builtin_memcpy(&data.daddr, args->daddr, sizeof(data.daddr));
__builtin_memcpy(&data.saddr_v6, args->saddr_v6, sizeof(data.saddr_v6));
__builtin_memcpy(&data.daddr_v6, args->daddr_v6, sizeof(data.daddr_v6));
bpf_perf_event_output(args, &tcprstsend_events, BPF_F_CURRENT_CPU, &data, sizeof(data));
return 0;
}
What I found was that this code compiles and can be loaded into the kernel, but fails when you are attaching it to the tracepoint.
It fails with a permission error stating that it can't be attached to the pfd. Here's the actual message
libbpf: program 'tracepoint/tcp/tcp_send_reset': failed to attach to pfd 92: Permission denied
libbpf: program 'tracepoint/tcp/tcp_send_reset': failed to attach to tracepoint 'tcp/tcp_send_reset': Permission denied
I switched from __builtin_memcpy to bpf_probe_read to see if that would help and it resolved the permission errors and allowed me to attach to the tracepoint, but I found that the data wasn't read correctly. The "state" member of the tcp_send_rst_args struct that I defined isn't included in CentOS 8/kernel 4.18 so all my reads were off by four bytes on CentOS. It works fine if I redefine that struct to:
struct tcp_send_rst_args {
long long pad;
const void * skbaddr;
const void * skaddr;
#ifndef RHEL_RELEASE_CODE
int state; // This needs to be removed for CentOS 8/Linux 4.18
#endif
u16 sport;
u16 dport;
u8 saddr[4];
u8 daddr[4];
u8 saddr_v6[16];
u8 daddr_v6[16];
};
Now I'm a bit confused because __builtin_memcpy seemed to fail at attach time rather than load time. However, it did actually fail (albeit with error messages that ended up being really hard to debug, I will never NOT check the tracepoint format file again though). bpf_probe_read just happily read past the struct.
I'm not sure where the memory it was reading was and if that should be defined behavior, but I thought I would send this here and see if this is intended or if I have actually found something unexpected. Should __builtin_memcpy be used? Or should bpf_probe_read? If bpf_probe_read is recommended, is there a way we can verify that we're not reading garbage data in this context other than having a human eyeball the data returned? Or is that just a necessary part of BPF development in this context? Is this issue something that the verifier can even check at load time? I can provide more information on the program and/or bug if it's needed, thanks!