__builtin_memcpy behavior


Tristan Mayfield
 

The other day I was in the process of porting a little libbpf application from Ubuntu 20 (Linux 5.4) to CentOS 8 (Linux 4.18). This program uses tracepoint:tcp:tcp_send_reset. Here's the relevant BPF code:

struct tcp_send_rst_args {
    long long pad;
    const void * skbaddr;
    const void * skaddr;
    int state;
    u16 sport;
    u16 dport;
    u8 saddr[4];
    u8 daddr[4];
    u8 saddr_v6[16];
    u8 daddr_v6[16];
};

SEC("tracepoint/tcp/tcp_send_reset")
int tcp_send_reset_prog(struct tcp_send_rst_args * args) {

    struct tcprstsend_data_t data = {};
    data.pid = bpf_get_current_pid_tgid() >> 32;

    data.sport = args->sport;
    data.dport = args->dport;
    bpf_get_current_comm(&data.comm, sizeof(data.comm));

    __builtin_memcpy(&data.saddr, args->saddr, sizeof(data.saddr));
    __builtin_memcpy(&data.daddr, args->daddr, sizeof(data.daddr));

    __builtin_memcpy(&data.saddr_v6, args->saddr_v6, sizeof(data.saddr_v6));
    __builtin_memcpy(&data.daddr_v6, args->daddr_v6, sizeof(data.daddr_v6));

    bpf_perf_event_output(args, &tcprstsend_events, BPF_F_CURRENT_CPU, &data, sizeof(data));
    return 0;
}

What I found was that this code compiles and can be loaded into the kernel, but fails when you are attaching it to the tracepoint.
It fails with a permission error stating that it can't be attached to the pfd. Here's the actual message

libbpf: program 'tracepoint/tcp/tcp_send_reset': failed to attach to pfd 92: Permission denied
libbpf: program 'tracepoint/tcp/tcp_send_reset': failed to attach to tracepoint 'tcp/tcp_send_reset': Permission denied

I switched from __builtin_memcpy to bpf_probe_read to see if that would help and it resolved the permission errors and allowed me to attach to the tracepoint, but I found that the data wasn't read correctly. The "state" member of the tcp_send_rst_args struct that I defined isn't included in CentOS 8/kernel 4.18 so all my reads were off by four bytes on CentOS. It works fine if I redefine that struct to:

struct tcp_send_rst_args {
    long long pad;
    const void * skbaddr;
    const void * skaddr;
#ifndef RHEL_RELEASE_CODE
    int state; // This needs to be removed for CentOS 8/Linux 4.18
#endif
    u16 sport;
    u16 dport;
    u8 saddr[4];
    u8 daddr[4];
    u8 saddr_v6[16];
    u8 daddr_v6[16];
};

Now I'm a bit confused because __builtin_memcpy seemed to fail at attach time rather than load time. However, it did actually fail (albeit with error messages that ended up being really hard to debug, I will never NOT check the tracepoint format file again though). bpf_probe_read just happily read past the struct.
I'm not sure where the memory it was reading was and if that should be defined behavior, but I thought I would send this here and see if this is intended or if I have actually found something unexpected. Should __builtin_memcpy be used? Or should bpf_probe_read? If bpf_probe_read is recommended, is there a way we can verify that we're not reading garbage data in this context other than having a human eyeball the data returned? Or is that just a necessary part of BPF development in this context? Is this issue something that the verifier can even check at load time? I can provide more information on the program and/or bug if it's needed, thanks!

Join {iovisor-dev@lists.iovisor.org to automatically receive all group messages.