Date   

Re: clang 10 for BPF CO-RE

Jesper Dangaard Brouer
 

On Wed, 11 Mar 2020 10:36:47 -0700
"Andrii Nakryiko" <andrii.nakryiko@...> wrote:

On Wed, Mar 11, 2020 at 10:33 AM <tmayfield@...> wrote:

Hi all,

Finally found the BPF blog and it's been nice to get more
information on using libbpf directly since I don't have a lot of
systems or kernel experience.
Thanks! Glad it was useful.
I assume this is the blog post[1]:
[1] https://facebookmicrosites.github.io/bpf/blog/2020/02/20/bcc-to-libbpf-howto-guide.html
Thanks for writing that Andrii!

For using libbpf directly from C, we also have the XDP-tutorial[2], but
doesn't contain BPF CO-RE examples. And it uses the old style map
defines. We are planning to update/fix that, once LLVM 10 gets more
widely available in distros.

[2] https://github.com/xdp-project/xdp-tutorial


Scanning through the "BCC to libbpf" post, I noticed Andrii calls
for using clang 10. I went to look at llvm releases and only saw
clang/llvm 9 (as of September 2019). Is clang 10 just built from
source?
For kernel/libbpf development we do build Clang from sources, but you
can install it from packages as well. See https://apt.llvm.org/, there
are packages for Clang 10 and even Clang 11 and they are updated
frequently.
Let me give you the manual compile recipe (that I got from Eelco):

git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir -p llvm/build/install
cd llvm/build
cmake -G "Ninja" -DLLVM_TARGETS_TO_BUILD="BPF;X86" \
-DLLVM_ENABLE_PROJECTS="clang" \
-DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PWD/install ..
ninja && ninja install
export PATH=$PWD/install/bin:$PATH

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer


Re: clang 10 for BPF CO-RE

Andrii Nakryiko
 

On Wed, Mar 11, 2020 at 10:33 AM <tmayfield@...> wrote:

Hi all,

Finally found the BPF blog and it's been nice to get more information on using libbpf directly since I don't have a lot of systems or kernel experience.
Thanks! Glad it was useful.


Scanning through the "BCC to libbpf" post, I noticed Andrii calls for using clang 10. I went to look at llvm releases and only saw clang/llvm 9 (as of September 2019).
Is clang 10 just built from source?
For kernel/libbpf development we do build Clang from sources, but you
can install it from packages as well. See https://apt.llvm.org/, there
are packages for Clang 10 and even Clang 11 and they are updated
frequently.


Looking forward to building with CO-RE and move some of my BCC tooling to libbpf.
Great, please do!


-Tristan


clang 10 for BPF CO-RE

Tristan Mayfield
 

Hi all,
 
Finally found the BPF blog and it's been nice to get more information on using libbpf directly since I don't have a lot of systems or kernel experience.
 
Scanning through the "BCC to libbpf" post, I noticed Andrii calls for using clang 10. I went to look at llvm releases and only saw clang/llvm 9 (as of September 2019).
Is clang 10 just built from source?
 
Looking forward to building with CO-RE and move some of my BCC tooling to libbpf.
 
-Tristan


Re: Getting function's address from BPF_TRACE_FENTRY BPF program

Yutaro Hayakawa
 

I see, so this means the fentry program
needs to load and verify the program for
every functions to attach right?

In my (maybe very specific) case, the
tool may attaches programs to more than
1000 functions. So it is important to
reduce the programs to reduce the attach
time.

I will continue to use kprobe. Thank you very
much for your help.

Yutaro

On Mar 8, 2020, at 4:19, Alexei Starovoitov <alexei.starovoitov@...> wrote:

On Fri, Mar 6, 2020 at 11:19 PM Yutaro Hayakawa <yhayakawa3720@...> wrote:

Hello,

Is there any way to get the address of the function in fentry type programs like
kprobe type programs does by PT_REGS_IP(pt_regs)?

I'd like to migrate my kprobe based tool[1] to fentry based one, but only this
feature is missing right now. Since the tool attaches single BPF program to
the multiple kernel functions, it needs to have function's address to identify
which function the trace data comes from.

[1] https://github.com/YutaroHayakawa/ipftrace
I think this approach won't quite work with fentry because
the same fenty type prog cannot be attached to multiple kernel functions.
At load time the kernel verifier needs to hold target kernel function,
check that arguments match, etc. So at that point the target function
address is fixed and when fentry prog is called it will see only one
'faddr' == regs_ip.


Re: Getting function's address from BPF_TRACE_FENTRY BPF program

Alexei Starovoitov
 

On Fri, Mar 6, 2020 at 11:19 PM Yutaro Hayakawa <yhayakawa3720@...> wrote:

Hello,

Is there any way to get the address of the function in fentry type programs like
kprobe type programs does by PT_REGS_IP(pt_regs)?

I'd like to migrate my kprobe based tool[1] to fentry based one, but only this
feature is missing right now. Since the tool attaches single BPF program to
the multiple kernel functions, it needs to have function's address to identify
which function the trace data comes from.

[1] https://github.com/YutaroHayakawa/ipftrace
I think this approach won't quite work with fentry because
the same fenty type prog cannot be attached to multiple kernel functions.
At load time the kernel verifier needs to hold target kernel function,
check that arguments match, etc. So at that point the target function
address is fixed and when fentry prog is called it will see only one
'faddr' == regs_ip.


Getting function's address from BPF_TRACE_FENTRY BPF program

Yutaro Hayakawa
 

Hello,

Is there any way to get the address of the function in fentry type programs like
kprobe type programs does by PT_REGS_IP(pt_regs)?

I'd like to migrate my kprobe based tool[1] to fentry based one, but only this
feature is missing right now. Since the tool attaches single BPF program to
the multiple kernel functions, it needs to have function's address to identify
which function the trace data comes from.

Regards,
Yutaro


why bpf output wakeup_events and sample_period is 1?

Hayden Livingston
 

wakeup_events and sample_period is set to 1. what is the reason for this?

Isn't it better if this number is higher so the polling doesn't happen
all the time?

what is "sample_period" if wakeup_events tells kernel to wake up.


Re: BCC integration into Buildroot

Jugurtha BELKALEM
 


Hi,
Have you looked at using libbpf and BPF CO-RE for such use cases? The difference is that you won't have any additional runtime dependencies (no Clang/LLVM, etc), which makes this more suitable for embedded applications. The main requirement for running BPF CO-RE programs would be to compile kernel with CONFIG_DEBUG_INFO_BTF=y for BTF type information. Check out also https://github.com/iovisor/bcc/pull/2755 that adds first BPF CO-RE converted tool to BCC. See few links below for more details.


Unfortunately, no; I have not used libbpf directly but I was thinking of doing it.

My goal for having BCC integrated into buildroot is is that embedded systems are not so limited as they were before. THis brings the following advantages : 
- We can reuse BCC scripts made for desktops and run them on embedded devices. BCC can fit smoothly to provide us with a clear, easy simple and ease script maintenance (easy even for non C developers who can understand quickly).
- Having python parsing returned results opens a bunch of endless possibilities like drawing graphs, saving to a remote database or even hand it them to an IA engine to understand system's behaviour over time (without having to develop another application for that).

People used SystemTap in the past for some embedded systems, some others are using LTTng for debugging. So why not BCC (though, It's right that we need more space compared to hard coded ebpf).

Thanks for your response, I'm going to try it out.

Regards.


Re: BCC integration into Buildroot

Andrii Nakryiko
 



On Mon, Jun 3, 2019 at 4:52 AM Jugurtha BELKALEM <jugurtha.belkalem@...> wrote:
Hi,

I've been doing some Linux debugging since one year, and I've used  BCC to solve multiple issues (like writting a ddos detector : https://github.com/iovisor/bcc/blob/master/examples/tracing/dddos.py). I have made an article : http://www.linuxembedded.fr/2019/03/les-secrets-du-traceur-ebpf/ (to present BCC to french community).

But, because my job focuses mainly on embedded systems; I and my colleague "Romain Naour" ported BCC to the Buildroot project and tests were already successful for ARM64 (Raspberry PI 3) as described in this article : http://www.linuxembedded.fr/2019/05/bcc-integration-into-buildroot/.

BCC is such a great tool and I'd love to know what you think about running it on tiny devices.

Have you looked at using libbpf and BPF CO-RE for such use cases? The difference is that you won't have any additional runtime dependencies (no Clang/LLVM, etc), which makes this more suitable for embedded applications. The main requirement for running BPF CO-RE programs would be to compile kernel with CONFIG_DEBUG_INFO_BTF=y for BTF type information. Check out also https://github.com/iovisor/bcc/pull/2755 that adds first BPF CO-RE converted tool to BCC. See few links below for more details.


 

Note : sorry if you have received this mail twice, I've just added the mailing list.
Regards. 

--

Jugurtha.


--
SMILE 

32 boulevard Vincent Gâche
44200 NANTES

Jugurtha BELKALEM
Ingénieur Etude et Développement 1


Twitter Facebook LinkedIn Github


eco Pour la planète, n'imprimez ce mail que si c'est nécessaire
                    
      


Re: Bcc for Android #bcc #android

Dale Hamel
 

I also have a WIP branch of bpftrace that supports bionic libc, for Android.

On Tue, Feb 25, 2020 at 07:03 Mingo <novelinuxer@...> wrote:
Does bcc have an adaptation plan for the Android platform?


Bcc for Android #bcc #android

Mingo
 

Does bcc have an adaptation plan for the Android platform?


Re: Run CO-RE version's runqslower failed

Andrii Nakryiko
 

On Sun, Feb 23, 2020 at 7:39 PM Andrii Nakryiko via Lists.Iovisor.Org
<andrii.nakryiko=gmail.com@...> wrote:

On Sun, Feb 23, 2020 at 6:52 PM <ethercflow@...> wrote:

[Edited Message Follows]

I tried to run CO-RE version's runqslower failed, the error info:

libbpf: sched_wakeup is not found in vmlinux BTF
libbpf: failed to load object 'runqslower_bpf'
libbpf: failed to load BPF skeleton 'runqslower_bpf': -2
failed to load BPF object: -2

ENV

clang version 10.0.0-+rc2-1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin


Linux Kernel: 5.6.0-rc2+ (commit 8eece07c011f88da0ccf4127fca9a4e4faaf58ae)

CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_TEST_BPF=m


CONFIG_VIDEO_SONY_BTF_MPX=m
CONFIG_DEBUG_INFO_BTF=y


With gdb's help, I found the `btf__find_by_name_kind` return -ENOENT.
I printed all name: https://transfer.sh/ANKNs/log and found btf_trace_sched_wakeup doesn't exist.

Hi!

runqslower expects that kernel was built with BTF type info (which is
enabled by CONFIG_DEBUG_INFO_BTF=y Kconfig option). Can you please
re-build your kernel with BTF enabled
and try again?
Discussion has been moved to https://github.com/iovisor/bcc/issues/2770




Re: Run CO-RE version's runqslower failed

Andrii Nakryiko
 

On Sun, Feb 23, 2020 at 6:52 PM <ethercflow@...> wrote:

[Edited Message Follows]

I tried to run CO-RE version's runqslower failed, the error info:

libbpf: sched_wakeup is not found in vmlinux BTF
libbpf: failed to load object 'runqslower_bpf'
libbpf: failed to load BPF skeleton 'runqslower_bpf': -2
failed to load BPF object: -2

ENV

clang version 10.0.0-+rc2-1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin


Linux Kernel: 5.6.0-rc2+ (commit 8eece07c011f88da0ccf4127fca9a4e4faaf58ae)

CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_TEST_BPF=m


CONFIG_VIDEO_SONY_BTF_MPX=m
CONFIG_DEBUG_INFO_BTF=y


With gdb's help, I found the `btf__find_by_name_kind` return -ENOENT.
I printed all name: https://transfer.sh/ANKNs/log and found btf_trace_sched_wakeup doesn't exist.

Hi!

runqslower expects that kernel was built with BTF type info (which is
enabled by CONFIG_DEBUG_INFO_BTF=y Kconfig option). Can you please
re-build your kernel with BTF enabled
and try again?



Run CO-RE version's runqslower failed

ethercflow@...
 
Edited

I tried to run CO-RE version's runqslower failed, the error info:

libbpf: sched_wakeup is not found in vmlinux BTF
libbpf: failed to load object 'runqslower_bpf'
libbpf: failed to load BPF skeleton 'runqslower_bpf': -2
failed to load BPF object: -2
ENV
clang version 10.0.0-+rc2-1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Linux Kernel: 5.6.0-rc2+ (commit 8eece07c011f88da0ccf4127fca9a4e4faaf58ae)
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_TEST_BPF=m

CONFIG_VIDEO_SONY_BTF_MPX=m
CONFIG_DEBUG_INFO_BTF=y

With gdb's help, I found the `btf__find_by_name_kind` return -ENOENT.
I printed all name: https://transfer.sh/ANKNs/log and found btf_trace_sched_wakeup doesn't exist. 




Confused about wakeup watermark vs sample period when attaching to BPF program

Hayden Livingston
 

Please correct me if I'm wrong about anything.

When a perf_event is attached to a BPF program and the BPF program is
going to do processing and then output what is the significant of
wakeup_events or wakeup_watermark for the original perf_event?

To me it seem like it BPF program will always run, but in the absence
of mmap buffer in original perf_event does it matter?

Then also, what should I set my BPF_OUTPUT wakeup to? Should I set to
large number? How can I get notified in my BPF OUTPUT (not the
original perf) after every 5 seconds? Is that possible?


Re: Can multiple BPF programs use same per-cpu perf ring buffer?

Yonghong Song
 

On Sun, Feb 16, 2020 at 8:43 PM Hayden Livingston
<halivingston@...> wrote:

Imagine I have a per-cpu perf ring buffer for all my cpus.

Now I have two eBPF programs.

In both these eBPF programs I do bpf_update_elem(myFD, &cpunumberkey,
&fdOfCPUspecificBuffer, BPF_ANY)

Will this mean that multiple eBPF programs will be able to write their
data into a single buffer (of course associated with cpu).

This would be amazing if it is truly possible. It seems like it should
be possible.
Yes, you can do this.


I have not tried yet.



Re: bpf_probe_read and pagefaults

Hayden Livingston
 

I should have search. Short answer it fails and you're out of luck.

https://lists.iovisor.org/g/iovisor-dev/topic/accessing_user_memory_and/21386221

On Sun, Feb 16, 2020 at 9:29 PM Hayden Livingston
<halivingston@...> wrote:

I'm curios to know how bpf_probe_read is able to read user-mode memory
in the face of page faulting.

I know in the helper it disables page faulting, but what does that mean?

If the memory the probe is trying to read is paged out then how does
my probe work?

It seems bpf_probe_read is best effort then. Is that true?


bpf_probe_read and pagefaults

Hayden Livingston
 

I'm curios to know how bpf_probe_read is able to read user-mode memory
in the face of page faulting.

I know in the helper it disables page faulting, but what does that mean?

If the memory the probe is trying to read is paged out then how does
my probe work?

It seems bpf_probe_read is best effort then. Is that true?


Can multiple BPF programs use same per-cpu perf ring buffer?

Hayden Livingston
 

Imagine I have a per-cpu perf ring buffer for all my cpus.

Now I have two eBPF programs.

In both these eBPF programs I do bpf_update_elem(myFD, &cpunumberkey,
&fdOfCPUspecificBuffer, BPF_ANY)

Will this mean that multiple eBPF programs will be able to write their
data into a single buffer (of course associated with cpu).

This would be amazing if it is truly possible. It seems like it should
be possible.

I have not tried yet.


Re: Why is BPF_PERF_OUTPUT max_entries set to total processor count?

Yonghong Song
 

On Sun, Feb 16, 2020 at 5:09 PM Hayden Livingston
<halivingston@...> wrote:

Thanks. I had to re-read your reply and the kernel code multiple
times, but I think I get it now. Please confirm.

It is this call is made by user mode code:

fd = bpf_create_map(BPF_MAP_TYPE_PERF_EVENT_ARRAY, /*key_size*/
sizeof(int), /*value_size*/ sizeof(int), NUM_POSSIBLE_CPUS, 0);

key is smp_processor_id. value is perf_events fd. This is why the map
is both is key integer and value integer.

Why so many indirections? Is it to support pinning where user program
can different ring buffers?
Perf event ring buffer is per cpu.


To me it seems the kernel code that uses cpu index to look into array
could just to told fd directly.
Yes, it is what it did in the kernel. Each array element holds one ring buffer.


On Sun, Feb 16, 2020 at 1:50 PM Y Song <ys114321@...> wrote:

PERF_EVENT_OUTPUT map is to hold per cpu ring buffers created by
perf_event_open.
That is why its typical size is the number of cpus on the host.

On Sun, Feb 16, 2020 at 1:52 AM Hayden Livingston
<halivingston@...> wrote:

I'm very confused why BCC creates a map of number of processors for
the perf_events output map.

I can imagine it being 1 since all it does is act as a kernel-user
mode intermediary and it is true that the code cannot be preempted.

Or if it can be preempted then I can imagine that since there can't be
more than processor count it is the max depth one has to worry about.

Is my thinking flawed? Or maybe there is a completely different reason?



201 - 220 of 2020