libbpf: sched_wakeup is not found in vmlinux BTF

libbpf: failed to load object 'runqslower_bpf'

libbpf: failed to load BPF skeleton 'runqslower_bpf': -2

failed to load BPF object: -2

ENV

clang version 10.0.0-+rc2-1

Target: x86_64-pc-linux-gnu

Thread model: posix

InstalledDir: /usr/bin

Linux Kernel: 5.6.0-rc2+ (commit 8eece07c011f88da0ccf4127fca9a4e4faaf58ae)

Please correct me if I'm wrong about anything.

When a perf_event is attached to a BPF program and the BPF program is
going to do processing and then output what is the significant of
wakeup_events or wakeup_watermark for the original perf_event?

To me it seem like it BPF program will always run, but in the absence
of mmap buffer in original perf_event does it matter?

Then also, what should I set my BPF_OUTPUT wakeup to? Should I set to
large number? How can I get notified in my BPF OUTPUT (not the
original perf) after every 5 seconds? Is that possible?

On Sun, Feb 16, 2020 at 8:43 PM Hayden Livingston
<halivingston@...> wrote:

Imagine I have a per-cpu perf ring buffer for all my cpus.

Now I have two eBPF programs.

In both these eBPF programs I do bpf_update_elem(myFD, &cpunumberkey,
&fdOfCPUspecificBuffer, BPF_ANY)

Will this mean that multiple eBPF programs will be able to write their
data into a single buffer (of course associated with cpu).

This would be amazing if it is truly possible. It seems like it should
be possible.

Yes, you can do this.

I have not tried yet.

I should have search. Short answer it fails and you're out of luck.

https://lists.iovisor.org/g/iovisor-dev/topic/accessing_user_memory_and/21386221

On Sun, Feb 16, 2020 at 9:29 PM Hayden Livingston
<halivingston@...> wrote:

I'm curios to know how bpf_probe_read is able to read user-mode memory
in the face of page faulting.

I know in the helper it disables page faulting, but what does that mean?

If the memory the probe is trying to read is paged out then how does
my probe work?

It seems bpf_probe_read is best effort then. Is that true?

I'm curios to know how bpf_probe_read is able to read user-mode memory
in the face of page faulting.

I know in the helper it disables page faulting, but what does that mean?

If the memory the probe is trying to read is paged out then how does
my probe work?

It seems bpf_probe_read is best effort then. Is that true?

Imagine I have a per-cpu perf ring buffer for all my cpus.

Now I have two eBPF programs.

In both these eBPF programs I do bpf_update_elem(myFD, &cpunumberkey,
&fdOfCPUspecificBuffer, BPF_ANY)

Will this mean that multiple eBPF programs will be able to write their
data into a single buffer (of course associated with cpu).

This would be amazing if it is truly possible. It seems like it should
be possible.

I have not tried yet.

On Sun, Feb 16, 2020 at 5:09 PM Hayden Livingston
<halivingston@...> wrote:

Thanks. I had to re-read your reply and the kernel code multiple
times, but I think I get it now. Please confirm.

It is this call is made by user mode code:

fd = bpf_create_map(BPF_MAP_TYPE_PERF_EVENT_ARRAY, /*key_size*/
sizeof(int), /*value_size*/ sizeof(int), NUM_POSSIBLE_CPUS, 0);

key is smp_processor_id. value is perf_events fd. This is why the map
is both is key integer and value integer.

Why so many indirections? Is it to support pinning where user program
can different ring buffers?

Perf event ring buffer is per cpu.

To me it seems the kernel code that uses cpu index to look into array
could just to told fd directly.

Yes, it is what it did in the kernel. Each array element holds one ring buffer.

On Sun, Feb 16, 2020 at 1:50 PM Y Song <ys114321@...> wrote:

PERF_EVENT_OUTPUT map is to hold per cpu ring buffers created by
perf_event_open.
That is why its typical size is the number of cpus on the host.

On Sun, Feb 16, 2020 at 1:52 AM Hayden Livingston
<halivingston@...> wrote:

I'm very confused why BCC creates a map of number of processors for
the perf_events output map.

I can imagine it being 1 since all it does is act as a kernel-user
mode intermediary and it is true that the code cannot be preempted.

Or if it can be preempted then I can imagine that since there can't be
more than processor count it is the max depth one has to worry about.

Is my thinking flawed? Or maybe there is a completely different reason?

Thanks. I had to re-read your reply and the kernel code multiple
times, but I think I get it now. Please confirm.

It is this call is made by user mode code:

fd = bpf_create_map(BPF_MAP_TYPE_PERF_EVENT_ARRAY, /*key_size*/
sizeof(int), /*value_size*/ sizeof(int), NUM_POSSIBLE_CPUS, 0);

key is smp_processor_id. value is perf_events fd. This is why the map
is both is key integer and value integer.

Why so many indirections? Is it to support pinning where user program
can different ring buffers?

To me it seems the kernel code that uses cpu index to look into array
could just to told fd directly.

PERF_EVENT_OUTPUT map is to hold per cpu ring buffers created by
perf_event_open.
That is why its typical size is the number of cpus on the host.

On Sun, Feb 16, 2020 at 1:52 AM Hayden Livingston
<halivingston@...> wrote:

I'm very confused why BCC creates a map of number of processors for
the perf_events output map.

I can imagine it being 1 since all it does is act as a kernel-user
mode intermediary and it is true that the code cannot be preempted.

Or if it can be preempted then I can imagine that since there can't be
more than processor count it is the max depth one has to worry about.

Is my thinking flawed? Or maybe there is a completely different reason?

I'm very confused why BCC creates a map of number of processors for
the perf_events output map.

I can imagine it being 1 since all it does is act as a kernel-user
mode intermediary and it is true that the code cannot be preempted.

Or if it can be preempted then I can imagine that since there can't be
more than processor count it is the max depth one has to worry about.

Is my thinking flawed? Or maybe there is a completely different reason?

Hello,

I was looking for a tool to collect latency on all IP connections through a linux router. We do have eBPF tools setup on the box. I am looking for something similar to tcpconnlat but need to capture the statistics for IP connections transiting through the router. 

Please suggest the best way to capture this. 

Best Regards,
Vignesh

There's no API to access command line args. BPF_FUNC_get_current_comm
will give you the task name. If it's not enough, you can try to get it
via task_struct. Look for get_task_cmdline fs/proc/base.c in the
Kernel source code as a starting point to get the cmdline from a
task_struct.

Hi all,

bcc monitoring tools which print a process being traced print only the command (and pid, ppid) without the full args. In many cases the monitored command is a script, so the command is just printed as (for example) "python" which isn't very useful. I couldn't find a bpf API to get the command line args.

Ganesan

Hi all, 

 

I'm trying to trace zfs reads and writes and collect statistics for different pools using the pool name.  

Here is a simplified version that just has a call count. 

kprobe:zfs_read,kprobe:zfs_write
{
	$spa_name = ((struct zfsvfs *) ((struct inode *)arg0)->i_sb->s_fs_info)->z_os->os_spa->spa_name;
	@zfs_count[$spa_name] = count();
}

I'm hitting the stack limit due to two large strings of length 256 being allocated on the stack: 

  %"@zfs_count_key" = alloca [256 x i8]
  %"$spa_name" = alloca [256 x i8]

0.9.3 is tagged: https://github.com/iovisor/bpftrace/releases/tag/v0.9.3 .

Please see changelog for more details.

Daniel

On Mon, 04 Nov 2019 13:42:32 -0800
"Daniel Xu" <dxu@...> wrote:

I was thinking about a 0.9.3 release and I noticed we had

https://github.com/iovisor/bpftrace/milestone/6

but it appears neglected.

Any thoughts on punting those issues and cutting a release now anyways?

We've merged a bunch of useful things (BTF, signed ints, int casts, better errors,
etc) since 0.9.2.

I would VERY much like to see a release, because I have bpftrace
scripts that depend on latest git-tree (I think it's the 'int casts'
that I depend on).

My scripts are here:
https://github.com/xdp-project/xdp-project/tree/master/areas/mem/bpftrace

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer

I'll update the 0.9.3 milestone and create a 0.9.4.

In the future I'll try to keep milestones updated for PRs and issues.

I guess this is also a good time to open the floor for other things that
should go into 0.9.3.

I was thinking about a 0.9.3 release and I noticed we had

https://github.com/iovisor/bpftrace/milestone/6

but it appears neglected.

Any thoughts on punting those issues and cutting a release now anyways?

We've merged a bunch of useful things (BTF, signed ints, int casts, better errors,
etc) since 0.9.2.

And yes, I am volunteering to drive the release.

On Wed, Oct 02, 2019 at 07:43:31PM +0200, Jiri Olsa wrote:

hi,
we'd like to have bcc linked with libbpf instead of the
github submodule, initial change is discussed in here:
https://github.com/iovisor/bcc/pull/2535

In order to do that, we need to have access to uapi headers
compatible with libbpf rpm, bcc is attaching and using them
during compilation.

I added them in the fedora spec below (not submitted yet),
so libbpf would carry those headers.

Thoughts? thanks,

I think it may break a bunch of people who rely on bcc being a single library.
What is the main motiviation to use libbpf as a shared library in libbcc?

I think we can have both options. libbpf as git submodule and as shared.
In practice git submodule is so much simpler to use and a lot less headaches.