Date   

Getting function's address from BPF_TRACE_FENTRY BPF program

Yutaro Hayakawa
 

Hello,

Is there any way to get the address of the function in fentry type programs like
kprobe type programs does by PT_REGS_IP(pt_regs)?

I'd like to migrate my kprobe based tool[1] to fentry based one, but only this
feature is missing right now. Since the tool attaches single BPF program to
the multiple kernel functions, it needs to have function's address to identify
which function the trace data comes from.

Regards,
Yutaro


why bpf output wakeup_events and sample_period is 1?

Hayden Livingston
 

wakeup_events and sample_period is set to 1. what is the reason for this?

Isn't it better if this number is higher so the polling doesn't happen
all the time?

what is "sample_period" if wakeup_events tells kernel to wake up.


Re: BCC integration into Buildroot

Jugurtha BELKALEM
 


Hi,
Have you looked at using libbpf and BPF CO-RE for such use cases? The difference is that you won't have any additional runtime dependencies (no Clang/LLVM, etc), which makes this more suitable for embedded applications. The main requirement for running BPF CO-RE programs would be to compile kernel with CONFIG_DEBUG_INFO_BTF=y for BTF type information. Check out also https://github.com/iovisor/bcc/pull/2755 that adds first BPF CO-RE converted tool to BCC. See few links below for more details.


Unfortunately, no; I have not used libbpf directly but I was thinking of doing it.

My goal for having BCC integrated into buildroot is is that embedded systems are not so limited as they were before. THis brings the following advantages : 
- We can reuse BCC scripts made for desktops and run them on embedded devices. BCC can fit smoothly to provide us with a clear, easy simple and ease script maintenance (easy even for non C developers who can understand quickly).
- Having python parsing returned results opens a bunch of endless possibilities like drawing graphs, saving to a remote database or even hand it them to an IA engine to understand system's behaviour over time (without having to develop another application for that).

People used SystemTap in the past for some embedded systems, some others are using LTTng for debugging. So why not BCC (though, It's right that we need more space compared to hard coded ebpf).

Thanks for your response, I'm going to try it out.

Regards.


Re: BCC integration into Buildroot

Andrii Nakryiko
 



On Mon, Jun 3, 2019 at 4:52 AM Jugurtha BELKALEM <jugurtha.belkalem@...> wrote:
Hi,

I've been doing some Linux debugging since one year, and I've used  BCC to solve multiple issues (like writting a ddos detector : https://github.com/iovisor/bcc/blob/master/examples/tracing/dddos.py). I have made an article : http://www.linuxembedded.fr/2019/03/les-secrets-du-traceur-ebpf/ (to present BCC to french community).

But, because my job focuses mainly on embedded systems; I and my colleague "Romain Naour" ported BCC to the Buildroot project and tests were already successful for ARM64 (Raspberry PI 3) as described in this article : http://www.linuxembedded.fr/2019/05/bcc-integration-into-buildroot/.

BCC is such a great tool and I'd love to know what you think about running it on tiny devices.

Have you looked at using libbpf and BPF CO-RE for such use cases? The difference is that you won't have any additional runtime dependencies (no Clang/LLVM, etc), which makes this more suitable for embedded applications. The main requirement for running BPF CO-RE programs would be to compile kernel with CONFIG_DEBUG_INFO_BTF=y for BTF type information. Check out also https://github.com/iovisor/bcc/pull/2755 that adds first BPF CO-RE converted tool to BCC. See few links below for more details.


 

Note : sorry if you have received this mail twice, I've just added the mailing list.
Regards. 

--

Jugurtha.


--
SMILE 

32 boulevard Vincent Gâche
44200 NANTES

Jugurtha BELKALEM
Ingénieur Etude et Développement 1


Twitter Facebook LinkedIn Github


eco Pour la planète, n'imprimez ce mail que si c'est nécessaire
                    
      


Re: Bcc for Android #bcc #android

Dale Hamel
 

I also have a WIP branch of bpftrace that supports bionic libc, for Android.

On Tue, Feb 25, 2020 at 07:03 Mingo <novelinuxer@...> wrote:
Does bcc have an adaptation plan for the Android platform?


Bcc for Android #bcc #android

Mingo
 

Does bcc have an adaptation plan for the Android platform?


Re: Run CO-RE version's runqslower failed

Andrii Nakryiko
 

On Sun, Feb 23, 2020 at 7:39 PM Andrii Nakryiko via Lists.Iovisor.Org
<andrii.nakryiko=gmail.com@...> wrote:

On Sun, Feb 23, 2020 at 6:52 PM <ethercflow@...> wrote:

[Edited Message Follows]

I tried to run CO-RE version's runqslower failed, the error info:

libbpf: sched_wakeup is not found in vmlinux BTF
libbpf: failed to load object 'runqslower_bpf'
libbpf: failed to load BPF skeleton 'runqslower_bpf': -2
failed to load BPF object: -2

ENV

clang version 10.0.0-+rc2-1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin


Linux Kernel: 5.6.0-rc2+ (commit 8eece07c011f88da0ccf4127fca9a4e4faaf58ae)

CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_TEST_BPF=m


CONFIG_VIDEO_SONY_BTF_MPX=m
CONFIG_DEBUG_INFO_BTF=y


With gdb's help, I found the `btf__find_by_name_kind` return -ENOENT.
I printed all name: https://transfer.sh/ANKNs/log and found btf_trace_sched_wakeup doesn't exist.

Hi!

runqslower expects that kernel was built with BTF type info (which is
enabled by CONFIG_DEBUG_INFO_BTF=y Kconfig option). Can you please
re-build your kernel with BTF enabled
and try again?
Discussion has been moved to https://github.com/iovisor/bcc/issues/2770




Re: Run CO-RE version's runqslower failed

Andrii Nakryiko
 

On Sun, Feb 23, 2020 at 6:52 PM <ethercflow@...> wrote:

[Edited Message Follows]

I tried to run CO-RE version's runqslower failed, the error info:

libbpf: sched_wakeup is not found in vmlinux BTF
libbpf: failed to load object 'runqslower_bpf'
libbpf: failed to load BPF skeleton 'runqslower_bpf': -2
failed to load BPF object: -2

ENV

clang version 10.0.0-+rc2-1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin


Linux Kernel: 5.6.0-rc2+ (commit 8eece07c011f88da0ccf4127fca9a4e4faaf58ae)

CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_TEST_BPF=m


CONFIG_VIDEO_SONY_BTF_MPX=m
CONFIG_DEBUG_INFO_BTF=y


With gdb's help, I found the `btf__find_by_name_kind` return -ENOENT.
I printed all name: https://transfer.sh/ANKNs/log and found btf_trace_sched_wakeup doesn't exist.

Hi!

runqslower expects that kernel was built with BTF type info (which is
enabled by CONFIG_DEBUG_INFO_BTF=y Kconfig option). Can you please
re-build your kernel with BTF enabled
and try again?



Run CO-RE version's runqslower failed

ethercflow@...
 
Edited

I tried to run CO-RE version's runqslower failed, the error info:

libbpf: sched_wakeup is not found in vmlinux BTF
libbpf: failed to load object 'runqslower_bpf'
libbpf: failed to load BPF skeleton 'runqslower_bpf': -2
failed to load BPF object: -2
ENV
clang version 10.0.0-+rc2-1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

Linux Kernel: 5.6.0-rc2+ (commit 8eece07c011f88da0ccf4127fca9a4e4faaf58ae)
CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_TEST_BPF=m

CONFIG_VIDEO_SONY_BTF_MPX=m
CONFIG_DEBUG_INFO_BTF=y

With gdb's help, I found the `btf__find_by_name_kind` return -ENOENT.
I printed all name: https://transfer.sh/ANKNs/log and found btf_trace_sched_wakeup doesn't exist. 




Confused about wakeup watermark vs sample period when attaching to BPF program

Hayden Livingston
 

Please correct me if I'm wrong about anything.

When a perf_event is attached to a BPF program and the BPF program is
going to do processing and then output what is the significant of
wakeup_events or wakeup_watermark for the original perf_event?

To me it seem like it BPF program will always run, but in the absence
of mmap buffer in original perf_event does it matter?

Then also, what should I set my BPF_OUTPUT wakeup to? Should I set to
large number? How can I get notified in my BPF OUTPUT (not the
original perf) after every 5 seconds? Is that possible?


Re: Can multiple BPF programs use same per-cpu perf ring buffer?

Yonghong Song
 

On Sun, Feb 16, 2020 at 8:43 PM Hayden Livingston
<halivingston@...> wrote:

Imagine I have a per-cpu perf ring buffer for all my cpus.

Now I have two eBPF programs.

In both these eBPF programs I do bpf_update_elem(myFD, &cpunumberkey,
&fdOfCPUspecificBuffer, BPF_ANY)

Will this mean that multiple eBPF programs will be able to write their
data into a single buffer (of course associated with cpu).

This would be amazing if it is truly possible. It seems like it should
be possible.
Yes, you can do this.


I have not tried yet.



Re: bpf_probe_read and pagefaults

Hayden Livingston
 

I should have search. Short answer it fails and you're out of luck.

https://lists.iovisor.org/g/iovisor-dev/topic/accessing_user_memory_and/21386221

On Sun, Feb 16, 2020 at 9:29 PM Hayden Livingston
<halivingston@...> wrote:

I'm curios to know how bpf_probe_read is able to read user-mode memory
in the face of page faulting.

I know in the helper it disables page faulting, but what does that mean?

If the memory the probe is trying to read is paged out then how does
my probe work?

It seems bpf_probe_read is best effort then. Is that true?


bpf_probe_read and pagefaults

Hayden Livingston
 

I'm curios to know how bpf_probe_read is able to read user-mode memory
in the face of page faulting.

I know in the helper it disables page faulting, but what does that mean?

If the memory the probe is trying to read is paged out then how does
my probe work?

It seems bpf_probe_read is best effort then. Is that true?


Can multiple BPF programs use same per-cpu perf ring buffer?

Hayden Livingston
 

Imagine I have a per-cpu perf ring buffer for all my cpus.

Now I have two eBPF programs.

In both these eBPF programs I do bpf_update_elem(myFD, &cpunumberkey,
&fdOfCPUspecificBuffer, BPF_ANY)

Will this mean that multiple eBPF programs will be able to write their
data into a single buffer (of course associated with cpu).

This would be amazing if it is truly possible. It seems like it should
be possible.

I have not tried yet.


Re: Why is BPF_PERF_OUTPUT max_entries set to total processor count?

Yonghong Song
 

On Sun, Feb 16, 2020 at 5:09 PM Hayden Livingston
<halivingston@...> wrote:

Thanks. I had to re-read your reply and the kernel code multiple
times, but I think I get it now. Please confirm.

It is this call is made by user mode code:

fd = bpf_create_map(BPF_MAP_TYPE_PERF_EVENT_ARRAY, /*key_size*/
sizeof(int), /*value_size*/ sizeof(int), NUM_POSSIBLE_CPUS, 0);

key is smp_processor_id. value is perf_events fd. This is why the map
is both is key integer and value integer.

Why so many indirections? Is it to support pinning where user program
can different ring buffers?
Perf event ring buffer is per cpu.


To me it seems the kernel code that uses cpu index to look into array
could just to told fd directly.
Yes, it is what it did in the kernel. Each array element holds one ring buffer.


On Sun, Feb 16, 2020 at 1:50 PM Y Song <ys114321@...> wrote:

PERF_EVENT_OUTPUT map is to hold per cpu ring buffers created by
perf_event_open.
That is why its typical size is the number of cpus on the host.

On Sun, Feb 16, 2020 at 1:52 AM Hayden Livingston
<halivingston@...> wrote:

I'm very confused why BCC creates a map of number of processors for
the perf_events output map.

I can imagine it being 1 since all it does is act as a kernel-user
mode intermediary and it is true that the code cannot be preempted.

Or if it can be preempted then I can imagine that since there can't be
more than processor count it is the max depth one has to worry about.

Is my thinking flawed? Or maybe there is a completely different reason?




Re: Why is BPF_PERF_OUTPUT max_entries set to total processor count?

Hayden Livingston
 

Thanks. I had to re-read your reply and the kernel code multiple
times, but I think I get it now. Please confirm.

It is this call is made by user mode code:

fd = bpf_create_map(BPF_MAP_TYPE_PERF_EVENT_ARRAY, /*key_size*/
sizeof(int), /*value_size*/ sizeof(int), NUM_POSSIBLE_CPUS, 0);

key is smp_processor_id. value is perf_events fd. This is why the map
is both is key integer and value integer.

Why so many indirections? Is it to support pinning where user program
can different ring buffers?

To me it seems the kernel code that uses cpu index to look into array
could just to told fd directly.

On Sun, Feb 16, 2020 at 1:50 PM Y Song <ys114321@...> wrote:

PERF_EVENT_OUTPUT map is to hold per cpu ring buffers created by
perf_event_open.
That is why its typical size is the number of cpus on the host.

On Sun, Feb 16, 2020 at 1:52 AM Hayden Livingston
<halivingston@...> wrote:

I'm very confused why BCC creates a map of number of processors for
the perf_events output map.

I can imagine it being 1 since all it does is act as a kernel-user
mode intermediary and it is true that the code cannot be preempted.

Or if it can be preempted then I can imagine that since there can't be
more than processor count it is the max depth one has to worry about.

Is my thinking flawed? Or maybe there is a completely different reason?



Re: Why is BPF_PERF_OUTPUT max_entries set to total processor count?

Yonghong Song
 

PERF_EVENT_OUTPUT map is to hold per cpu ring buffers created by
perf_event_open.
That is why its typical size is the number of cpus on the host.

On Sun, Feb 16, 2020 at 1:52 AM Hayden Livingston
<halivingston@...> wrote:

I'm very confused why BCC creates a map of number of processors for
the perf_events output map.

I can imagine it being 1 since all it does is act as a kernel-user
mode intermediary and it is true that the code cannot be preempted.

Or if it can be preempted then I can imagine that since there can't be
more than processor count it is the max depth one has to worry about.

Is my thinking flawed? Or maybe there is a completely different reason?



Why is BPF_PERF_OUTPUT max_entries set to total processor count?

Hayden Livingston
 

I'm very confused why BCC creates a map of number of processors for
the perf_events output map.

I can imagine it being 1 since all it does is act as a kernel-user
mode intermediary and it is true that the code cannot be preempted.

Or if it can be preempted then I can imagine that since there can't be
more than processor count it is the max depth one has to worry about.

Is my thinking flawed? Or maybe there is a completely different reason?


ebpf Tool to collect latency on all IP connections through a Linux router

vignesh_ramamurthy@...
 

Hello,

I was looking for a tool to collect latency on all IP connections through a linux router. We do have eBPF tools setup on the box. I am looking for something similar to tcpconnlat but need to capture the statistics for IP connections transiting through the router. 

Please suggest the best way to capture this. 

Best Regards,
Vignesh


Re: Is there an API to get the process command line?

Ganesan Rajagopal
 

Thanks Quillian. I considered tracing sys_execve since execsnoop already provides sample code for that. I also need to trace process exits to remove the pid to command line mapping. This is a very busy build server and spawning processes like crazy, so keeping a live mapping of all the processes and command lines may be too resource intensive. I'll give it a shot and see how it goes.

Ganesan

On Fri, Jan 3, 2020 at 1:58 AM Quillian Rutherford <quillian.rutherford@...> wrote:
If you are running while the process is created, you can set an entry probe on sys_execve and it has the cmdline in the arguments.  probe like:

int enter_sys_execve(struct pt_regs *ctx,
  const char __user *filename,
  const char __user *const __user *__argv,
  const char __user *const __user *__envp){


Then you can submit back the contents of argv.

On Wed, Jan 1, 2020 at 7:56 AM <rganesan+iovisor@...> wrote:
Hi all,

bcc monitoring tools which print a process being traced print only the command (and pid, ppid) without the full args. In many cases the monitored command is a script, so the command is just printed as (for example) "python" which isn't very useful. I couldn't find a bpf API to get the command line args.

Ganesan

201 - 220 of 2015