Date   

Re: Extracting data from tracepoints (and anything else)

Andrii Nakryiko
 

On Mon, Mar 23, 2020 at 9:38 AM <mayfieldtristan@...> wrote:

I've been exploring the libbpf library for different versions of the Linux kernel, and trying to rewrite some of the BCC tools. I would like to do more work with CO-RE eventually, but I'm trying to understand the entire model of how BPF programs work and how data flows between the kernel, the VM, and userspace. I just started using perf buffers instead of bpf_trace_printk and came across an issue that has me scratching my head. In the below code, I'm not able to access the const char * arg in the tracepoint sys_enter_openat (kernel 4.15). For some reason the verifier rejects this code. I think it's valid C (although I'm a little bit rusty still) and I think I followed the correct flow where data must be copied from the kernel to the VM before being able to use.

If anyone has insight to share, I'd much appreciate it. Conversely, if anyone can point me in the direction of how to debug BPF programs that would be extremely helpful too. Should I just dig into learning the basics of BPF asm?

Highlights of the code:

struct bpf_map_def SEC("maps") events = {
.type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
.key_size = sizeof(int),
.value_size = sizeof(u32),
.max_entries = MAX_CPUS,
};
nit: this is a legacy syntax of specifying BPF maps, please see [0]
for some newer examples

[0] https://github.com/iovisor/bcc/tree/master/libbpf-tools


struct sys_enter_openat_args {
u16 common_type;
u8 common_flags;
u8 common_preempt_count;
int common_pid;
int __syscall_nr;
int dfd;
char *filename;
int flags;
__mode_t mode;
};

SEC("tracepoint/syscalls/sys_enter_openat")
int bpf_prog(struct sys_enter_openat_args *ctx) {
struct data_t data;
struct sys_enter_openat_args *args;

int res = bpf_probe_read(args, sizeof(ctx), ctx);
you don't need to bpf_probe_read() ctx here, you can just access its
members directly.

if(!res) {
data.file = "couldn't get file";
} else {
data.file = args->filename;
But here if you want to read filename contents itself, you'll need to
use bpf_probe_read_str().

Having data_t definition would be also helpful.

}

Error Message:

bpf_load_program() err=13
0: (bf) r6 = r1
1: (b7) r2 = 8
2: (bf) r3 = r6
3: (85) call bpf_probe_read#4
R1 type=ctx expected=fp
this error from verifier is quite misleading, but what verifier
complains about here is that you try to read uninitialized pointer
(arg) and pass it as a first parameter into bpf_probe_read(). But see
above, you don't need to bpf_probe_read() anything, and even if you
wanted to it would have to be done very differently:

struct sys_enter_openat_args args; /* notice no pointer here */
bpf_probe_read(&args, sizeof(args), ctx); /* taking address of args,
taking size of args, not its pointer */

The kernel didn't load the BPF program

data.pid = bpf_get_current_pid_tgid(); // use fn from libbpf.h to get pid_tgid
bpf_get_current_comm(data.program_name, sizeof(data.program_name)); // puts current comm into char array

bpf_perf_event_output(ctx, &events, 0, &data, sizeof(data));

return 0;
}

If more code would be helpful, I'm happy to share.

I recognize that libbpf and CO-RE in later kernels provides an easier API for dealing with char * (bpf_probe_read_str() I believe) but I'm trying to understand what needs to be done to target different kernels and not just the most cutting edge.

As a second question, how much should I learn about perf(1) and its overlap with BPF?

Finally, for long-term monitoring solutions and passing readable data, do most programs rely on pinning maps to the vfs instead of using perf buffers or passing directly to a userspace process?
It's a mix. If your data should/can be pre-aggregated in kernel, using
map might benefit you in that you will be sending much less data to
user-space. But if you want to send every piece of information than
perf_buffer is faster and more convenient than having user-space query
BPF maps all the time.


Thanks for the patience and goodwill with a new systems dev. I've enjoyed my interactions with the BPF community.
You're welcome. Check libbpf-tools in BCC repo, it should give you
some examples to work off of.


Tristan


Array brace-enclosed initialization

Federico Parola <fede.parola@...>
 

Hello everybody,
in my XDP eBPF program I'm trying to initialize an array with a brace-enclosed list, however my code is rejected by the verifier.
Here is a simple piece of code to replicate the problem:

#include <linux/bpf.h>

#ifndef __section
# define __section(NAME)                  \
   __attribute__((section(NAME), used))
#endif

#ifndef BPF_FUNC
# define BPF_FUNC(NAME, ...)              \
   (*NAME)(__VA_ARGS__) = (void *)BPF_FUNC_##NAME
#endif

#ifndef printk
# define printk(fmt, ...)                                      \
    ({                                                         \
        char ____fmt[] = fmt;                                  \
        trace_printk(____fmt, sizeof(____fmt), ##__VA_ARGS__); \
    })
#endif

static void BPF_FUNC(trace_printk, const char *fmt, int fmt_size, ...);

__section("prog")
int xdp_prog(struct xdp_md *ctx) {
  int i;
  int array[10] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9};

#pragma nounroll
  for (i = 0; i < 10; i++) {
    printk("%d", array[i]);
  }

  return XDP_PASS;
}

char __license[] __section("license") = "GPL"
This is the error reported by the verifier:
0: (b7) r6 = 0
1: (b7) r7 = 25637
2: (b7) r8 = 0
3: (73) *(u8 *)(r10 -2) = r6
last_idx 3 first_idx 0
regs=40 stack=0 before 2: (b7) r8 = 0
regs=40 stack=0 before 1: (b7) r7 = 25637
regs=40 stack=0 before 0: (b7) r6 = 0
4: (6b) *(u16 *)(r10 -4) = r7
5: (18) r1 = 0x0
7: (0f) r1 += r8
8: (61) r3 = *(u32 *)(r1 +0)
R1 invalid mem access 'inv'
processed 8 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
I tried compiling with clang-6 and clang-9 with optimizaton set to O1 and O2, but I get this error in all cases.
If I initialize the array in another way (e.g. with a loop) the program works correctly.
The eBPF bytecode generated by clang is the following:
	.text
	.file	"test.c"
	.section	prog,"ax",@progbits
	.globl	xdp_prog                # -- Begin function xdp_prog
	.p2align	3
	.type	xdp_prog,@function
xdp_prog:                               # @xdp_prog
# %bb.0:
	r6 = 0
	r7 = 25637
	r8 = 0
LBB0_1:                                 # =>This Inner Loop Header: Depth=1
	*(u8 *)(r10 - 2) = r6
	*(u16 *)(r10 - 4) = r7
	r1 = .L__const.xdp_prog.array ll
	r1 += r8
	r3 = *(u32 *)(r1 + 0)
	r1 = r10
	r1 += -4
	r2 = 3
	call 6
	r8 += 4
	if r8 != 24 goto LBB0_1
# %bb.2:
	r0 = 2
	exit
.Lfunc_end0:
	.size	xdp_prog, .Lfunc_end0-xdp_prog
                                        # -- End function
	.type	.L__const.xdp_prog.array,@object # @__const.xdp_prog.array
	.section	.rodata,"a",@progbits
	.p2align	2
.L__const.xdp_prog.array:
	.long	0                       # 0x0
	.long	1                       # 0x1
	.long	2                       # 0x2
	.long	3                       # 0x3
	.long	4                       # 0x4
	.long	5                       # 0x5
	.long	6                       # 0x6
	.long	7                       # 0x7
	.long	8                       # 0x8
	.long	9                       # 0x9
	.size	.L__const.xdp_prog.array, 40

	.type	.L__const.xdp_prog.____fmt,@object # @__const.xdp_prog.____fmt
	.section	.rodata.str1.1,"aMS",@progbits,1
.L__const.xdp_prog.____fmt:
	.asciz	"%d"
	.size	.L__const.xdp_prog.____fmt, 3

	.type	__license,@object       # @__license
	.section	license,"aw",@progbits
	.globl	__license
__license:
	.asciz	"GPL"
	.size	__license, 4


	.addrsig
	.addrsig_sym xdp_prog
	.addrsig_sym __license
It seems like the array in the stack is not initialized in the code. With some declarations the code works, for example decalring the array in the following way:
int array[10] = {0, 1, 2, 3};
everything works, the generated bytecode is the following:
	.text
	.file	"test.c"
	.section	prog,"ax",@progbits
	.globl	xdp_prog                # -- Begin function xdp_prog
	.p2align	3
	.type	xdp_prog,@function
xdp_prog:                               # @xdp_prog
# %bb.0:
	r1 = 2
	*(u32 *)(r10 - 36) = r1
	r6 = 0
	*(u32 *)(r10 - 8) = r6
	*(u32 *)(r10 - 12) = r6
	*(u32 *)(r10 - 16) = r6
	*(u32 *)(r10 - 20) = r6
	*(u32 *)(r10 - 24) = r6
	*(u32 *)(r10 - 28) = r6
	*(u32 *)(r10 - 4) = r6
	r1 = 3
	*(u32 *)(r10 - 32) = r1
	r1 = 1
	*(u32 *)(r10 - 40) = r1
	*(u8 *)(r10 - 42) = r6
	r7 = 25637
	*(u16 *)(r10 - 44) = r7
	r1 = r10
	r1 += -44
	r2 = 3
	r3 = 1
	call 6
	r8 = 4
LBB0_1:                                 # =>This Inner Loop Header: Depth=1
	r1 = r10
	r1 += -40
	r1 += r8
	r3 = *(u32 *)(r1 + 0)
	*(u8 *)(r10 - 42) = r6
	*(u16 *)(r10 - 44) = r7
	r1 = r10
	r1 += -44
	r2 = 3
	call 6
	r8 += 4
	if r8 != 24 goto LBB0_1
# %bb.2:
	r0 = 2
	exit
.Lfunc_end0:
	.size	xdp_prog, .Lfunc_end0-xdp_prog
                                        # -- End function
	.type	.L__const.xdp_prog.____fmt,@object # @__const.xdp_prog.____fmt
	.section	.rodata.str1.1,"aMS",@progbits,1
.L__const.xdp_prog.____fmt:
	.asciz	"%d"
	.size	.L__const.xdp_prog.____fmt, 3

	.type	__license,@object       # @__license
	.section	license,"aw",@progbits
	.globl	__license
__license:
	.asciz	"GPL"
	.size	__license, 4


	.addrsig
	.addrsig_sym xdp_prog
	.addrsig_sym __license
This time the array is correctly initialized into code.
Is this a clang bug?


Extracting data from tracepoints (and anything else)

Tristan Mayfield
 

I've been exploring the libbpf library for different versions of the Linux kernel, and trying to rewrite some of the BCC tools. I would like to do more work with CO-RE eventually, but I'm trying to understand the entire model of how BPF programs work and how data flows between the kernel, the VM, and userspace. I just started using perf buffers instead of bpf_trace_printk and came across an issue that has me scratching my head. In the below code, I'm not able to access the const char * arg in the tracepoint sys_enter_openat (kernel 4.15). For some reason the verifier rejects this code. I think it's valid C (although I'm a little bit rusty still) and I think I followed the correct flow where data must be copied from the kernel to the VM before being able to use.

If anyone has insight to share, I'd much appreciate it. Conversely, if anyone can point me in the direction of how to debug BPF programs that would be extremely helpful too. Should I just dig into learning the basics of BPF asm?

Highlights of the code:

struct bpf_map_def SEC("maps") events = {
  .type = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
  .key_size = sizeof(int),
  .value_size = sizeof(u32),
  .max_entries = MAX_CPUS,
};

struct sys_enter_openat_args {
        u16 common_type;
        u8 common_flags;
        u8 common_preempt_count;
        int common_pid;
        int __syscall_nr;
        int dfd;
        char *filename;
        int flags;
        __mode_t mode;
};

SEC("tracepoint/syscalls/sys_enter_openat")
int bpf_prog(struct sys_enter_openat_args *ctx) {
  struct data_t data;
  struct sys_enter_openat_args *args;

  int res = bpf_probe_read(args, sizeof(ctx), ctx);
  if(!res) {
         data.file = "couldn't get file";
  } else {
         data.file = args->filename;
  }

Error Message:

bpf_load_program() err=13
0: (bf) r6 = r1
1: (b7) r2 = 8
2: (bf) r3 = r6
3: (85) call bpf_probe_read#4
R1 type=ctx expected=fp
The kernel didn't load the BPF program

  data.pid = bpf_get_current_pid_tgid(); // use fn from libbpf.h to get pid_tgid
  bpf_get_current_comm(data.program_name, sizeof(data.program_name)); // puts current comm into char array

  bpf_perf_event_output(ctx, &events, 0, &data, sizeof(data));

  return 0;
}

If more code would be helpful, I'm happy to share.

I recognize that libbpf and CO-RE in later kernels provides an easier API for dealing with char * (bpf_probe_read_str() I believe) but I'm trying to understand what needs to be done to target different kernels and not just the most cutting edge.

As a second question, how much should I learn about perf(1) and its overlap with BPF?

Finally, for long-term monitoring solutions and passing readable data, do most programs rely on pinning maps to the vfs instead of using perf buffers or passing directly to a userspace process?

Thanks for the patience and goodwill with a new systems dev. I've enjoyed my interactions with the BPF community.

Tristan


Study on annotation of design and implementation choices, and of technical debt

a.serebrenik@...
 

Dear all,

As software engineering research teams at the University of Sannio (Italy) and Eindhoven University of Technology (The Netherlands) we are interested in investigating the protocol used by developers while they have to annotate implementation and design choices during their normal development activities. More specifically, we are looking at whether, where and what kind of annotations developers usually use trying to be focused more on those annotations mainly aimed at highlighting that the code is not in the right shape (e.g., comments for annotating delayed or intended work activities such as TODO, FIXME, hack, workaround, etc). In the latter case, we are looking at what is the content of the above annotations, as well as how they usually behave while evolving the code that has been previously annotated.

When answering the survey, in case your annotation practices are different in different open source projects you may contribute, please refer to how you behave for the projects where you have been contacted.

Filling out the survey will take about 5 minutes.

Please note that your identity and personal data will not be disclosed, while we plan to use the aggregated results and anonymized responses as part of a scientific publication. 

If you have any questions about the questionnaire or our research, please do not hesitate to contact us.

You can find the survey link here:


Thanks and regards,

Gianmarco Fucci (gianmarcofucci94@...)
Fiorella Zampetti (fzampetti@...)
Alexander Serebrenik (a.serebrenik@...)
Massimiliano Di Penta (dipenta@...)

--


Re: is BCC tools safe to enable root privilegies in production?

Cristian Spinetta
 

Thanks for your fast reply!

In our infrastructure the owners of the app can logging into the production VMs that are running their apps and execute a restricted list of command with sudo (e.g. tcpdump, netstat, ...). The idea is to give root access to each script of bcc tool (all within /usr/share/bcc/tools/*). We are concerned if there are some bcc scripts that can run another command like in the example above or if there are other security concerns to be aware of.

Best,
Cristian Spinetta


On Fri, Mar 13, 2020 at 1:23 PM Brendan Gregg <brendan.d.gregg@...> wrote:
On Fri, Mar 13, 2020 at 7:59 AM Cristian Spinetta <cebspinetta@...> wrote:
>
> Hi all!
>
> I am curious about whether it is safe to enable root access to BCC scripts on production machines.
> In my company, each team has access to their instances via ssh, and we are thinking to allow them to use bcc in production. For this purpose we need to allow root access to any BCC tool. Do you think it would be safe? for example, is there some tool that can receive a command to execute? in that case it would be unsafe because someone could execute any command thought a bcc tool.
>
> e.g.:
> sudo /usr/share/bcc/tools/some-great-tool.sh dd if=/dev/zero of=/dev/sda bs=512 count=1 conv=notrunc

^^^^

sudo isn't safe. If you remove the BCC tool from this one-liner,
you'll find it still destroys your disk.

In practice the production concern I have is for the overhead of each
tool, hence the overhead section in each tool's man page.

Brendan

>
> Best,
> Cristian Spinetta
>




Re: is BCC tools safe to enable root privilegies in production?

Brendan Gregg
 

On Fri, Mar 13, 2020 at 7:59 AM Cristian Spinetta <cebspinetta@...> wrote:

Hi all!

I am curious about whether it is safe to enable root access to BCC scripts on production machines.
In my company, each team has access to their instances via ssh, and we are thinking to allow them to use bcc in production. For this purpose we need to allow root access to any BCC tool. Do you think it would be safe? for example, is there some tool that can receive a command to execute? in that case it would be unsafe because someone could execute any command thought a bcc tool.

e.g.:
sudo /usr/share/bcc/tools/some-great-tool.sh dd if=/dev/zero of=/dev/sda bs=512 count=1 conv=notrunc
^^^^

sudo isn't safe. If you remove the BCC tool from this one-liner,
you'll find it still destroys your disk.

In practice the production concern I have is for the overhead of each
tool, hence the overhead section in each tool's man page.

Brendan


Best,
Cristian Spinetta


is BCC tools safe to enable root privilegies in production?

Cristian Spinetta
 

Hi all!

I am curious about whether it is safe to enable root access to BCC scripts on production machines.
In my company, each team has access to their instances via ssh, and we are thinking to allow them to use bcc in production. For this purpose we need to allow root access to any BCC tool. Do you think it would be safe? for example, is there some tool that can receive a command to execute? in that case it would be unsafe because someone could execute any command thought a bcc tool.

e.g.:
sudo /usr/share/bcc/tools/some-great-tool.sh dd if=/dev/zero of=/dev/sda bs=512 count=1 conv=notrunc

Best,
Cristian Spinetta


Re: clang 10 for BPF CO-RE

Jesper Dangaard Brouer
 

On Wed, 11 Mar 2020 10:36:47 -0700
"Andrii Nakryiko" <andrii.nakryiko@...> wrote:

On Wed, Mar 11, 2020 at 10:33 AM <tmayfield@...> wrote:

Hi all,

Finally found the BPF blog and it's been nice to get more
information on using libbpf directly since I don't have a lot of
systems or kernel experience.
Thanks! Glad it was useful.
I assume this is the blog post[1]:
[1] https://facebookmicrosites.github.io/bpf/blog/2020/02/20/bcc-to-libbpf-howto-guide.html
Thanks for writing that Andrii!

For using libbpf directly from C, we also have the XDP-tutorial[2], but
doesn't contain BPF CO-RE examples. And it uses the old style map
defines. We are planning to update/fix that, once LLVM 10 gets more
widely available in distros.

[2] https://github.com/xdp-project/xdp-tutorial


Scanning through the "BCC to libbpf" post, I noticed Andrii calls
for using clang 10. I went to look at llvm releases and only saw
clang/llvm 9 (as of September 2019). Is clang 10 just built from
source?
For kernel/libbpf development we do build Clang from sources, but you
can install it from packages as well. See https://apt.llvm.org/, there
are packages for Clang 10 and even Clang 11 and they are updated
frequently.
Let me give you the manual compile recipe (that I got from Eelco):

git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir -p llvm/build/install
cd llvm/build
cmake -G "Ninja" -DLLVM_TARGETS_TO_BUILD="BPF;X86" \
-DLLVM_ENABLE_PROJECTS="clang" \
-DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$PWD/install ..
ninja && ninja install
export PATH=$PWD/install/bin:$PATH

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer


Re: clang 10 for BPF CO-RE

Andrii Nakryiko
 

On Wed, Mar 11, 2020 at 10:33 AM <tmayfield@...> wrote:

Hi all,

Finally found the BPF blog and it's been nice to get more information on using libbpf directly since I don't have a lot of systems or kernel experience.
Thanks! Glad it was useful.


Scanning through the "BCC to libbpf" post, I noticed Andrii calls for using clang 10. I went to look at llvm releases and only saw clang/llvm 9 (as of September 2019).
Is clang 10 just built from source?
For kernel/libbpf development we do build Clang from sources, but you
can install it from packages as well. See https://apt.llvm.org/, there
are packages for Clang 10 and even Clang 11 and they are updated
frequently.


Looking forward to building with CO-RE and move some of my BCC tooling to libbpf.
Great, please do!


-Tristan


clang 10 for BPF CO-RE

Tristan Mayfield
 

Hi all,
 
Finally found the BPF blog and it's been nice to get more information on using libbpf directly since I don't have a lot of systems or kernel experience.
 
Scanning through the "BCC to libbpf" post, I noticed Andrii calls for using clang 10. I went to look at llvm releases and only saw clang/llvm 9 (as of September 2019).
Is clang 10 just built from source?
 
Looking forward to building with CO-RE and move some of my BCC tooling to libbpf.
 
-Tristan


Re: Getting function's address from BPF_TRACE_FENTRY BPF program

Yutaro Hayakawa
 

I see, so this means the fentry program
needs to load and verify the program for
every functions to attach right?

In my (maybe very specific) case, the
tool may attaches programs to more than
1000 functions. So it is important to
reduce the programs to reduce the attach
time.

I will continue to use kprobe. Thank you very
much for your help.

Yutaro

On Mar 8, 2020, at 4:19, Alexei Starovoitov <alexei.starovoitov@...> wrote:

On Fri, Mar 6, 2020 at 11:19 PM Yutaro Hayakawa <yhayakawa3720@...> wrote:

Hello,

Is there any way to get the address of the function in fentry type programs like
kprobe type programs does by PT_REGS_IP(pt_regs)?

I'd like to migrate my kprobe based tool[1] to fentry based one, but only this
feature is missing right now. Since the tool attaches single BPF program to
the multiple kernel functions, it needs to have function's address to identify
which function the trace data comes from.

[1] https://github.com/YutaroHayakawa/ipftrace
I think this approach won't quite work with fentry because
the same fenty type prog cannot be attached to multiple kernel functions.
At load time the kernel verifier needs to hold target kernel function,
check that arguments match, etc. So at that point the target function
address is fixed and when fentry prog is called it will see only one
'faddr' == regs_ip.


Re: Getting function's address from BPF_TRACE_FENTRY BPF program

Alexei Starovoitov
 

On Fri, Mar 6, 2020 at 11:19 PM Yutaro Hayakawa <yhayakawa3720@...> wrote:

Hello,

Is there any way to get the address of the function in fentry type programs like
kprobe type programs does by PT_REGS_IP(pt_regs)?

I'd like to migrate my kprobe based tool[1] to fentry based one, but only this
feature is missing right now. Since the tool attaches single BPF program to
the multiple kernel functions, it needs to have function's address to identify
which function the trace data comes from.

[1] https://github.com/YutaroHayakawa/ipftrace
I think this approach won't quite work with fentry because
the same fenty type prog cannot be attached to multiple kernel functions.
At load time the kernel verifier needs to hold target kernel function,
check that arguments match, etc. So at that point the target function
address is fixed and when fentry prog is called it will see only one
'faddr' == regs_ip.


Getting function's address from BPF_TRACE_FENTRY BPF program

Yutaro Hayakawa
 

Hello,

Is there any way to get the address of the function in fentry type programs like
kprobe type programs does by PT_REGS_IP(pt_regs)?

I'd like to migrate my kprobe based tool[1] to fentry based one, but only this
feature is missing right now. Since the tool attaches single BPF program to
the multiple kernel functions, it needs to have function's address to identify
which function the trace data comes from.

Regards,
Yutaro


why bpf output wakeup_events and sample_period is 1?

Hayden Livingston
 

wakeup_events and sample_period is set to 1. what is the reason for this?

Isn't it better if this number is higher so the polling doesn't happen
all the time?

what is "sample_period" if wakeup_events tells kernel to wake up.


Re: BCC integration into Buildroot

Jugurtha BELKALEM
 


Hi,
Have you looked at using libbpf and BPF CO-RE for such use cases? The difference is that you won't have any additional runtime dependencies (no Clang/LLVM, etc), which makes this more suitable for embedded applications. The main requirement for running BPF CO-RE programs would be to compile kernel with CONFIG_DEBUG_INFO_BTF=y for BTF type information. Check out also https://github.com/iovisor/bcc/pull/2755 that adds first BPF CO-RE converted tool to BCC. See few links below for more details.


Unfortunately, no; I have not used libbpf directly but I was thinking of doing it.

My goal for having BCC integrated into buildroot is is that embedded systems are not so limited as they were before. THis brings the following advantages : 
- We can reuse BCC scripts made for desktops and run them on embedded devices. BCC can fit smoothly to provide us with a clear, easy simple and ease script maintenance (easy even for non C developers who can understand quickly).
- Having python parsing returned results opens a bunch of endless possibilities like drawing graphs, saving to a remote database or even hand it them to an IA engine to understand system's behaviour over time (without having to develop another application for that).

People used SystemTap in the past for some embedded systems, some others are using LTTng for debugging. So why not BCC (though, It's right that we need more space compared to hard coded ebpf).

Thanks for your response, I'm going to try it out.

Regards.


Re: BCC integration into Buildroot

Andrii Nakryiko
 



On Mon, Jun 3, 2019 at 4:52 AM Jugurtha BELKALEM <jugurtha.belkalem@...> wrote:
Hi,

I've been doing some Linux debugging since one year, and I've used  BCC to solve multiple issues (like writting a ddos detector : https://github.com/iovisor/bcc/blob/master/examples/tracing/dddos.py). I have made an article : http://www.linuxembedded.fr/2019/03/les-secrets-du-traceur-ebpf/ (to present BCC to french community).

But, because my job focuses mainly on embedded systems; I and my colleague "Romain Naour" ported BCC to the Buildroot project and tests were already successful for ARM64 (Raspberry PI 3) as described in this article : http://www.linuxembedded.fr/2019/05/bcc-integration-into-buildroot/.

BCC is such a great tool and I'd love to know what you think about running it on tiny devices.

Have you looked at using libbpf and BPF CO-RE for such use cases? The difference is that you won't have any additional runtime dependencies (no Clang/LLVM, etc), which makes this more suitable for embedded applications. The main requirement for running BPF CO-RE programs would be to compile kernel with CONFIG_DEBUG_INFO_BTF=y for BTF type information. Check out also https://github.com/iovisor/bcc/pull/2755 that adds first BPF CO-RE converted tool to BCC. See few links below for more details.


 

Note : sorry if you have received this mail twice, I've just added the mailing list.
Regards. 

--

Jugurtha.


--
SMILE 

32 boulevard Vincent Gâche
44200 NANTES

Jugurtha BELKALEM
Ingénieur Etude et Développement 1


Twitter Facebook LinkedIn Github


eco Pour la planète, n'imprimez ce mail que si c'est nécessaire
                    
      


Re: Bcc for Android #bcc #android

Dale Hamel
 

I also have a WIP branch of bpftrace that supports bionic libc, for Android.

On Tue, Feb 25, 2020 at 07:03 Mingo <novelinuxer@...> wrote:
Does bcc have an adaptation plan for the Android platform?


Bcc for Android #bcc #android

Mingo
 

Does bcc have an adaptation plan for the Android platform?


Re: Run CO-RE version's runqslower failed

Andrii Nakryiko
 

On Sun, Feb 23, 2020 at 7:39 PM Andrii Nakryiko via Lists.Iovisor.Org
<andrii.nakryiko=gmail.com@...> wrote:

On Sun, Feb 23, 2020 at 6:52 PM <ethercflow@...> wrote:

[Edited Message Follows]

I tried to run CO-RE version's runqslower failed, the error info:

libbpf: sched_wakeup is not found in vmlinux BTF
libbpf: failed to load object 'runqslower_bpf'
libbpf: failed to load BPF skeleton 'runqslower_bpf': -2
failed to load BPF object: -2

ENV

clang version 10.0.0-+rc2-1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin


Linux Kernel: 5.6.0-rc2+ (commit 8eece07c011f88da0ccf4127fca9a4e4faaf58ae)

CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_TEST_BPF=m


CONFIG_VIDEO_SONY_BTF_MPX=m
CONFIG_DEBUG_INFO_BTF=y


With gdb's help, I found the `btf__find_by_name_kind` return -ENOENT.
I printed all name: https://transfer.sh/ANKNs/log and found btf_trace_sched_wakeup doesn't exist.

Hi!

runqslower expects that kernel was built with BTF type info (which is
enabled by CONFIG_DEBUG_INFO_BTF=y Kconfig option). Can you please
re-build your kernel with BTF enabled
and try again?
Discussion has been moved to https://github.com/iovisor/bcc/issues/2770




Re: Run CO-RE version's runqslower failed

Andrii Nakryiko
 

On Sun, Feb 23, 2020 at 6:52 PM <ethercflow@...> wrote:

[Edited Message Follows]

I tried to run CO-RE version's runqslower failed, the error info:

libbpf: sched_wakeup is not found in vmlinux BTF
libbpf: failed to load object 'runqslower_bpf'
libbpf: failed to load BPF skeleton 'runqslower_bpf': -2
failed to load BPF object: -2

ENV

clang version 10.0.0-+rc2-1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin


Linux Kernel: 5.6.0-rc2+ (commit 8eece07c011f88da0ccf4127fca9a4e4faaf58ae)

CONFIG_CGROUP_BPF=y
CONFIG_BPF=y
CONFIG_BPF_SYSCALL=y
CONFIG_ARCH_WANT_DEFAULT_BPF_JIT=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT_DEFAULT_ON=y
CONFIG_IPV6_SEG6_BPF=y
CONFIG_NETFILTER_XT_MATCH_BPF=m
CONFIG_BPFILTER=y
CONFIG_BPFILTER_UMH=m
CONFIG_NET_CLS_BPF=m
CONFIG_NET_ACT_BPF=m
CONFIG_BPF_JIT=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_LWTUNNEL_BPF=y
CONFIG_HAVE_EBPF_JIT=y
CONFIG_BPF_EVENTS=y
CONFIG_BPF_KPROBE_OVERRIDE=y
CONFIG_TEST_BPF=m


CONFIG_VIDEO_SONY_BTF_MPX=m
CONFIG_DEBUG_INFO_BTF=y


With gdb's help, I found the `btf__find_by_name_kind` return -ENOENT.
I printed all name: https://transfer.sh/ANKNs/log and found btf_trace_sched_wakeup doesn't exist.

Hi!

runqslower expects that kernel was built with BTF type info (which is
enabled by CONFIG_DEBUG_INFO_BTF=y Kconfig option). Can you please
re-build your kernel with BTF enabled
and try again?


201 - 220 of 2027