"Tristan Mayfield" <mayfieldtristan@...> writes:

Toke, thanks for the quick response!

Yes, I was checking the bpf_probe_read return values, and was reading
the number of bytes expected, so nothing wrong there!

Right, in that case that's probably just because the struct in question
is next to some other valid memory (not sure where tracepoints keep
their data, but if it's on the stack, for instance, you'll have no
problem reading past it).

Now that you mention CO-RE, it does actually make sense that these
sorts of errors could be shifted to load time rather than attach time
(that the right phrase?). I've fiddled with CO-RE a bit but I haven't
adopted it for a few reasons (which I could certainly be mistaken
about).

I'm by no means the leading authority on CO-RE, but I can give answering
a shot; hopefully someone will chime in to correct me if I'm wrong :)

I don't have control over kernel versions or compilation flags for the
kernel on the systems I'm targeting and I've had significant
difficulty trying to compile CO-RE programs (e.g. from the BCC repo's
libbpf-tools) on Linux <5.4 because I've had a hard time getting the
vmlinux. I can't remember if I used bpftool though (this was about a
year ago that I last played with CO-RE), so perhaps I'll give it
another shot.

Yeah, getting all your ducks in a row when compiling can be a bit of an
issue. However, I don't think you need anything special from the kernel
at compile-time if you just compile your own programs with a vmlinux.h
file you generated on a kernel that has been compiled with BTF.

I've also been very unclear, and have gotten many different answers
regarding the target systems and whether they need to be custom
compiled with BTF enabled for CO-RE programs to run on them, or if you
can put a CO-RE program onto a generic kernel build and it "just
works?" From your answer, the answer seems to be that
/sys/kernel/btf/vmlinux needs to be on the target system, so it must
have that BTF_ENABLE flag set?

Well, you'll need the BTF information of the running kernel. It doesn't
*have* to come from /sys/kernel/btf/vmlinux, libbpf will look for it in
a few other locations as well:

https://github.com/libbpf/libbpf/blob/master/src/btf.c#L4583

Distros have gotten pretty good about enabling BTF in their kernel
builds, though, so it's getting increasingly feasible to rely on it. It
should certainly be available on RHEL8 (and thus CentOS 8).

If that's set, do you also need a vmlinux.h file as well? A coworker
was recently messing with CO-RE and seemed to think that deploying a
CO-RE program required shipping the vmlinux.h file and I think he
mentioned that file was about 1Gb big, which is certainly a no-go for
our position.

No, you don't need to ship the vmlinux.h file. That's just a regular
header file with an unusual amount of definitions in it, that will be
used at compile time. It can be useful to include a copy of it in your
source code repository, though, as mentioned above. That's what BCC
does, for instance:
https://github.com/iovisor/bcc/tree/master/libbpf-tools/x86

An no, it's not 1GB in size. Maybe that size was from before BTF
de-duplication got implemented? The one linked above is 2.7M.

In addition to that, I've been unclear in the role of BTF in BPF
generally. When I began tinkering with BPF I was under the impression
that BTF was *only* something used for CO-RE programs (something I
actually might've gotten from the article referenced and written by
Andrii), but I've periodically seen errors arise that cite BTF reasons
for erroring.

One common cause for this has been when loading 'tc' programs with
iproute2, because the iproute2 loader doesn't understand BTF and will
complain about it. That is usually harmless, though, but I agree it's
quite annoying. Fortunately, iproute2 has recently gained support for
using libbpf for its BPF loading, so hopefully that particular error
should go away before too long.

Unfortunately I haven't saved any of these errors and
can't remember the causes specifically, but something like the
"updated" maps declarations, i.e.

struct {
__uint(type, BPF_MAP_TYPE_PERF_EVENT_ARRAY);
__uint(key_size, sizeof(u32));
__uint(value_size, sizeof(u32));
} events SEC(".maps");

I've learned does use BTF?

Yes, the new-style map definitions use BTF. While BTF is ostensibly a
type format (i.e., something that describes C data types), Andrii
figured out that it is also possible to use it as a general purpose
key/value store. You do this by being a bit clever about how you
represent your data, which is what the __uint() macro in the above is
doing (it's encoding the integer value as the size of an array, which
becomes part of the type and thus embedded in the BTF). When loading,
libbpf will parse this data back out of the BTF data and use it when
creating the map. So you'll need BTF support in your compiler and in
libbpf to use this style of map definitions.

Am I misunderstanding what BTF is and the role it plays in BPF? Or
maybe has libbpf development moved so far toward CO-RE that non-CO-RE
development gets similar or the same error messages that just aren't
as clear for it?

Hmm, no, CO-RE is the specific feature that does relocations of struct
fields based on member names. This relies on BTF, but it's not the only
thing BTF is used for. The map definition is another, as you discovered,
and there are some program types that cannot work without BTF
information at all. Also, things like bpftool being able to print out
the struct layout of map values is using BTF. So you're certainly right
that the BPF ecosystem in general is moving towards using BTF in more
and more places. And I guess you're also right that this leads to some
cryptic error messages sometimes... :)

-Toke