iovisor-dev@lists.iovisor.org | bpftrace ustack() pid packing

Daniel Xu

#1688

I'm reading through codegen_llvm.cpp[0] and I noticed that ustack
map keys are packed with a PID. From what I understand, if you run

    bpftrace -e 'uprobe:/lib64/libc.so.6:usleep /comm == "a.out"/ { @[ustack(2)] = count() }'

and a.out has forked, it would cause duplicate stack traces to be outputted.
This has caused some confusion in the past with users.

I'm wondering what the intention behind PID packing is.

Thanks,
Daniel
---
[0]: https://github.com/iovisor/bpftrace/blob/master/src/ast/codegen_llvm.cpp#L748-L756

Brendan Gregg

#1689

On Tue, May 28, 2019 at 11:35 AM Daniel Xu <dxu@...> wrote:

I'm reading through codegen_llvm.cpp[0] and I noticed that ustack
map keys are packed with a PID. From what I understand, if you run

    bpftrace -e 'uprobe:/lib64/libc.so.6:usleep /comm == "a.out"/ { @[ustack(2)] = count() }'

and a.out has forked, it would cause duplicate stack traces to be outputted.
This has caused some confusion in the past with users.

I'm wondering what the intention behind PID packing is.
The stack ID returned by ustack() points to a stack as a list of
addresses. That gets translated sometime later in user-space by
bpftrace. At that point you just have an address, and you need to know
which PID it belongs to, so you can lookup that processes's address
mappings.

If BPF one day can save the stack as a list of strings in the kernel
(meaning the kernel must support user-space symbol lookup), then we
wouldn't need the PID (this has been discussed before).

I don't know about this fork problem you mention.

Brendan




Thanks,
Daniel
---
[0]: https://github.com/iovisor/bpftrace/blob/master/src/ast/codegen_llvm.cpp#L748-L756

Daniel Xu

#1701

Thanks Brendan, that makes sense.

Here's an example that can trigger the fork issue:

$ cat ustack_fork_dup.cpp
#include <iostream>
#include <unistd.h>
#include <stdio.h>

__attribute__((noinline)) void do_sleep() {
  ::usleep(500000);
}

__attribute__((noinline)) void pathAAAA() {
  do_sleep();
}

__attribute__((noinline)) void pathAAA() {
  pathAAAA();
}

__attribute__((noinline)) void pathAA() {
  pathAAA();
}

__attribute__((noinline)) void pathA() {
  pathAA();
}


int main() {
  ::fork();

  while (1) {
    pathA();
  }
}

$ sudo ./build/src/bpftrace -e 'uprobe:/lib64/libc.so.6:usleep /comm == "a.out"/ { @[ustack(2)] = count() }'                                                        
Attaching 1 probe...
^C

@[
    usleep+0
    pathAAAA()+9
]: 16
@[
    usleep+0
    pathAAAA()+9
]: 16

The user would expect only a single entry in the map.

Daniel

Show quoted text

On Tue, May 28, 2019, at 1:01 PM, Brendan Gregg wrote:
On Tue, May 28, 2019 at 11:35 AM Daniel Xu <dxu@...> wrote:

I'm reading through codegen_llvm.cpp[0] and I noticed that ustack
map keys are packed with a PID. From what I understand, if you run

    bpftrace -e 'uprobe:/lib64/libc.so.6:usleep /comm == "a.out"/ { @[ustack(2)] = count() }'

and a.out has forked, it would cause duplicate stack traces to be outputted.
This has caused some confusion in the past with users.

I'm wondering what the intention behind PID packing is.
The stack ID returned by ustack() points to a stack as a list of
addresses. That gets translated sometime later in user-space by
bpftrace. At that point you just have an address, and you need to know
which PID it belongs to, so you can lookup that processes's address
mappings.

If BPF one day can save the stack as a list of strings in the kernel
(meaning the kernel must support user-space symbol lookup), then we
wouldn't need the PID (this has been discussed before).

I don't know about this fork problem you mention.

Brendan




Thanks,
Daniel
---
[0]: https://github.com/iovisor/bpftrace/blob/master/src/ast/codegen_llvm.cpp#L748-L756