Date   

LPC 2021 Networking and BPF Track CFP (Reminder)

Daniel Borkmann
 

This is a reminder for the Call for Proposals (CFP) for the Networking and
BPF track at the 2021 edition of the Linux Plumbers Conference (LPC), which
will be held virtually on the wider Internet, on September 20th - 24th, 2021.

This year's Networking and BPF track technical committee is comprised of:

David S. Miller <davem@...>
Jakub Kicinski <kuba@...>
Eric Dumazet <edumazet@...>
Alexei Starovoitov <ast@...>
Daniel Borkmann <daniel@...>
Andrii Nakryiko <andrii@...>

We are seeking proposals of 40 minutes in length (including Q&A discussion),
optionally accompanied by papers of 2 to 10 pages in length.

Any kind of advanced Linux networking and/or BPF related topic will be considered.

Please submit your proposals through the official LPC website at:

https://linuxplumbersconf.org/event/11/abstracts/

Make sure to select "Networking & BPF Summit" in the Track pull-down menu.

Proposals must be submitted by August 13th, and submitters will be notified of
acceptance by August 16th.

Final slides and papers (as PDF) are due on the first day of the conference.


Using XDP in docker swarm to track outgoing traffic

Sebastião Santos Boavida Amaro
 

Hi everyone,
I am trying to use XDP to track outgoing traffic from docker containers deployed using docker swarm and running in a network using the driver overlay. I am using a simple xdp program based on [1], and I run this program on the network namespace of the container using nsenter and attach it to its eth0.
However, I am only able to detect the incoming packets and not the outgoing ones. When running tcpdump on the container network namespace I can see both incoming and outgoing packets. So I am a bit confused as to why XDP would not detect the outgoing ones.
Does anyone know why the reason for this or a general idea as to why this might happen?

[1]https://github.com/iovisor/bcc/blob/master/examples/networking/xdp/xdp_drop_count.py

Best Regards,
Sebastião Amaro


BPF map pinning

Raga lahari
 

Hi,

Can someone please help me to pin a BPF map in custom path for a TC program. My requirement is pin  a map in interface-specific path (like /sys/fs/bpf/eth0/)


Regards,
Ragalahari


Re: Question about map.increment() #bcc

Donald Hunter
 

On Sun, 25 Apr 2021 at 20:18, Y Song <ys114321@...> wrote:
This is a good question. In earlier bpf days, the key MUST be from
stack. Otherwise, the verifier will fail. Nowadays, things become
better and keys can be from verifier recognizable memory regions
(stack, key, value, allocated_mem, etc.). I think rewriter can be made
smart to check if the first argument of the increment is actually a
variable (instead of an expression), we can directly take the address
of it since the variable can be allocated on stack. The relevant code
is at b_frontend_action.cc. Do you want to take a look to see whether
you could help improve the bcc rewriter for this particular issue?
I am happy to take a look at the code and see if I can improve it at all.

Thanks,
Donald.


Re: Question about map.increment() #bcc

Yonghong Song
 

On Thu, Apr 22, 2021 at 4:18 AM Donald Hunter <donald.hunter@...> wrote:

Is there a reason why map.increment() internally copies the key into a stack variable? When building a key inline, it uses double the stack space and incurs the cost of a copy. For u64 keys this is fine but for larger custom keys, e.g. containing a char[] it blows up the stack pretty quickly.
This is a good question. In earlier bpf days, the key MUST be from
stack. Otherwise, the verifier will fail. Nowadays, things become
better and keys can be from verifier recognizable memory regions
(stack, key, value, allocated_mem, etc.). I think rewriter can be made
smart to check if the first argument of the increment is actually a
variable (instead of an expression), we can directly take the address
of it since the variable can be allocated on stack. The relevant code
is at b_frontend_action.cc. Do you want to take a look to see whether
you could help improve the bcc rewriter for this particular issue?


Thanks, Donald.


Question about map.increment() #bcc

Donald Hunter
 

Is there a reason why map.increment() internally copies the key into a stack variable? When building a key inline, it uses double the stack space and incurs the cost of a copy. For u64 keys this is fine but for larger custom keys, e.g. containing a char[] it blows up the stack pretty quickly.

Thanks, Donald.


Re: [libbpf] Questions about XDP/TC

Toke Høiland-Jørgensen
 

chenhengqi@... writes:

1. How do I attach `BPF_PROG_TYPE_SCHED_CLS`/`classifier` BPF programs to specific data path(i.e. ingress or egress) using libbpf ?
libbpf does not yet support attaching to TC hooks, but there is work in
progress to add this. See
https://lore.kernel.org/bpf/20210325120020.236504-4-memxor@gmail.com/

(an updated version should hopefully show up soon).

I found some comments related in the source:
```
The **BPF_F_INGRESS** value in *flags* is used to make the distinction (ingress path is selected if the flag is present, egress path otherwise).
```

How can I get that flag, am I missing something ?

2. How do I attach `XDP` BPF programs using specific mode(i.e.
xdpgeneric/xdpdrv)?
Just set XDP_FLAGS_SKB_MODE or XDP_FLAGS_DRV_MODE when attaching...

-Toke


[libbpf] Questions about XDP/TC

chenhengqi@...
 

1. How do I attach `BPF_PROG_TYPE_SCHED_CLS`/`classifier` BPF programs to specific data path(i.e. ingress or egress) using libbpf ?
 
I found some comments related in the source:
```
The **BPF_F_INGRESS** value in *flags* is used to make the distinction (ingress path is selected if the flag is present, egress path otherwise).
```
 
How can I get that flag, am I missing something ?
 
2. How do I attach `XDP` BPF programs using specific mode(i.e. xdpgeneric/xdpdrv)?
 
I tried googling but most of them lead to tc/ip tools. Thanks in advance.


Re: libffi trampolines and stack traces? : was Overly brief stack traces for Java/linux ?

Bradley Schatz
 

Thanks for your help with this - and for the sketch of how to proceed with it.

I'm going to look into the feasibility of fixing the frame pointer in the trampoline before attempting what you have outlined.

Cheers!


On 18/4/21, 3:37 am, "Y Song" <ys114321@...> wrote:

On Fri, Apr 16, 2021 at 12:56 PM Yonghong Song via lists.iovisor.org
<ys114321=gmail.com@...> wrote:
>
> On Thu, Apr 15, 2021 at 11:09 PM Bradley Schatz
> <bradley@...> wrote:
> >
> > I still digging into this issue, and have hacked memleak/bcc to show addresses when they cant be resolved. In other places, I'm seeing jit complied java stack frames showing up alongside C ones, which is the expected behaviour when using Java with -XX:+PreserveFramePointer.
> >
> > I'm still seeing these allocation traces with only two frames per the below.
> >
> > 11534336 bytes in 11 allocations from stack
> > [7efeebb115d4 unknown] [jna7632521838566054573.tmp ]
> > [7efec27f7440 unknown] [perf-15445.map]
> > 14680064 bytes in 14 allocations from stack
> > [7efeebb115d4 unknown] [jna7632521838566054573.tmp ]
> > [7efec27f7380 unknown] [perf-15445.map]
> >
> > I suspect the lower frame above is a libffi generated trampoline. Are there any known issues with the eBPF stack tracing infrastructure and such trampolines? Workarounds?
>
> The bpf stack unwinder is using the kernel one which is the frame
> pointer based and it may have issues with generated trampoline code
> which may mess up frame pointer based stack chain. One possibility is
> to use perf call-graph "dwarf" mode to get the raw data to user space
> and use more powerful library like libunwind etc. to unwind the stack.
> But I haven't do this together with bpf program yet, and cannot
> describe whether and how to do "dwarf" call-graph with bpf program
> together.

I briefly looked at the perf and kernel code and experimented with
perf dwarf mode.
It is possible for bpf to copy user stack to user space. You could do
the following:
- when you do perf event open, the sample_type needs to be set
properly. The following
is what callchain dwarf mode had:
sample_type = PERF_SAMPLE_IP|PERF_SAMPLE_TID|PERF_SAMPLE_TIME|
PERF_SAMPLE_ADDR|PERF_SAMPLE_CALLCHAIN|PERF_SAMPLE_CPU|
PERF_SAMPLE_PERIOD|PERF_SAMPLE_REGS_USER|
PERF_SAMPLE_STACK_USER|PERF_SAMPLE_DATA_SRC

I do not think you need all of them, you definitely need PERF_SAMPLE_REGS_USER,
maybe you can PERF_SAMPLE_RAW and PERF_SAMPLE_REGS_USER
to see whether it works or not.

In kernel header file linux:include/linux/perf_event.h, we have
struct perf_sample_data {
...
struct perf_regs regs_user;
struct perf_regs regs_intr;
u64 stack_user_size;
}

If you have access to perf_sample_data, you can read regs_user and find
stack pointer and then you can copy some bytes (say 2KB) to user space
for analysis. Using perf_event_output may be too expensive. perf is using
user mmap memory. You can also use a map for this purpose.

How to access perf_sample_data?
Typically sampling bpf program has the signature like
int do_sample(struct bpf_perf_event_data *ctx)
During program run, the program really got the following ctx
struct bpf_perf_event_data_kern {
bpf_user_pt_regs_t *regs;
struct perf_sample_data *data;
struct perf_event *event;
};
So you can use bpf_probe_read_kernel() to get 'data' and then other
information inside perf_sample_data.

You can experiment this way if you really like to see whether it can
solve your problem or not. If it works, I guess we can add a bpf helper
to do the copy as such copy length tends to be big.

>
> >
> > Thanks,
> > Bradley
> >
> >
> >
> > On 7/4/21, 9:04 pm, "Bradley Schatz" <bradley@...> wrote:
> >
> > > What does '[unknown] [perf-18047.map]' mean? Does this mean
> > > perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
> > > symbolization won't be possible. Maybe you want to double check this?
> >
> > The file perf-18047.map is there and from other parts of the stack trace I can see it being used to successfully resolve symbols.
> >
> > Thanks!
> >
> >
> >
> > On 7/4/21, 4:44 am, "Y Song" <ys114321@...> wrote:
> >
> > On Mon, Apr 5, 2021 at 10:08 PM Bradley Schatz
> > <bradley@...> wrote:
> > >
> > > Thanks for the suggestion. I found a tunable to keep the JNI shared library in memory after loading. As you can see below, it is no longer showing as deleted.
> > >
> > > 13238272 bytes in 404 allocations from stack
> > > [unknown] [jna2576903844543447777.tmp]
> > > [unknown] [perf-18047.map]
> >
> > I have no experience with perf-map-agent, but the following is what I guess:
> > [perf-18047.map] is used to find the mapping between address and symbol.
> > What does '[unknown] [perf-18047.map]' mean? Does this mean
> > perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
> > symbolization won't be possible. Maybe you want to double check this?
> >
> > >
> > > No improvement in granularity though.
> > >
> > > In the VM I'm using -XX:+PreserveFramePointer -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints. In perf_maps_agent, I'm using "unfoldall"
> > >
> > > Any other suggestions?
> > >
> > > Thanks!
> > >
> > >
> > >
> > >
> > > On 3/4/21, 2:42 am, "Y Song" <ys114321@...> wrote:
> > >
> > > On Wed, Mar 31, 2021 at 11:25 PM Bradley Schatz
> > > <bradley@...> wrote:
> > > >
> > > > Hi,
> > > >
> > > >
> > > >
> > > > I’m just starting to come to grips with bcc & perf-map-agent for introspecting java on linux, with the goal of identifying what appears to be an off-heap memory leak (using memleak).
> > > >
> > > >
> > > >
> > > > I appear to be getting reliable stack decoding for jvm library code and for jit’ed java methods (see below for an example of the former). However I am seeing some very short stack traces which don’t seem to decode (the latter three stacks of below).
> > > >
> > > >
> > > >
> > > > It’s looking to me like the frame starting with “jna…” is likely the native JNI shared library for the FFI library “JNA”.
> > > >
> > > >
> > > >
> > > > Any suggestions as to why these latter three are so brief and/or how I can increase the resolution?
> > >
> > > I can see the file has been marked as deleted.
> > >
> > > 34603008 bytes in 33 allocations from stack
> > >
> > > [unknown] [jna9005484735610534564.tmp (deleted)]
> > >
> > > [unknown] [perf-31566.map]
> > >
> > > 96468992 bytes in 92 allocations from stack
> > >
> > > [unknown] [jna9005484735610534564.tmp (deleted)]
> > >
> > > [unknown] [perf-31566.map]
> > >
> > > So the file has been removed in userspace and current bcc won't be
> > > pass to parse it since it takes the file name as
> > > "jna9005484735610534564.tmp (deleted)"
> > > The file name is actually taken from /proc/<pid>/maps.
> > >
> > > I am not sure whether you can hack to parse "jna9005484735610534564.tmp" or not.
> > > But I would consider it is unsafe to do that as the original file
> > > related info may just
> > > exist in kernel and there is a reference to it. For user space, it is
> > > either gone or
> > > could be replaced by something else. So the safest way is to find a place to
> > > do symbolization before file is gone or keep tmp file a little bit longer.
> > > >
> > > >
> > > >
> > > > Apologies if this is the wrong place for such a question. Thank you for your help.
> > > >
> > > >
> > > >
> > > > Kind regards,
> > > >
> > > > Bradley
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 119408 bytes in 71 allocations from stack
> > > >
> > > > os::malloc(unsigned long, MemoryType, NativeCallStack const&)+0xb5 [libjvm.so]
> > > >
> > > > CodeBlob::set_oop_maps(OopMapSet*) [clone .part.5]+0x75 [libjvm.so]
> > > >
> > > > CodeBlob::CodeBlob(char const*, CodeBuffer*, int, int, int, int, OopMapSet*)+0xe3 [libjvm.so]
> > > >
> > > > nmethod::nmethod(Method*, int, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x4d [libjvm.so]
> > > >
> > > > nmethod::new_nmethod(methodHandle, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x219 [libjvm.so]
> > > >
> > > > ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool, RTMState)+0x1b1 [libjvm.so]
> > > >
> > > > Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0xe60 [libjvm.so]
> > > >
> > > > C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa3 [libjvm.so]
> > > >
> > > > CompileBroker::invoke_compiler_on_method(CompileTask*)+0x808 [libjvm.so]
> > > >
> > > > CompileBroker::compiler_thread_loop()+0x6d8 [libjvm.so]
> > > >
> > > > JavaThread::thread_main_inner()+0x1c7 [libjvm.so]
> > > >
> > > > JavaThread::run()+0x2fa [libjvm.so]
> > > >
> > > > java_start(Thread*)+0x102 [libjvm.so]
> > > >
> > > > start_thread+0xf3 [libpthread-2.28.so]
> > > >
> > > > 34603008 bytes in 33 allocations from stack
> > > >
> > > > [unknown] [jna9005484735610534564.tmp (deleted)]
> > > >
> > > > [unknown] [perf-31566.map]
> > > >
> > > > 96468992 bytes in 92 allocations from stack
> > > >
> > > > [unknown] [jna9005484735610534564.tmp (deleted)]
> > > >
> > > > [unknown] [perf-31566.map]
> > > >
> > > > 295698432 bytes in 282 allocations from stack
> > > >
> > > > [unknown] [jna9005484735610534564.tmp (deleted)]
> > > >
> > > > [unknown] [perf-31566.map]
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
>
>
>
>
>


Re: libffi trampolines and stack traces? : was Overly brief stack traces for Java/linux ?

Yonghong Song
 

On Fri, Apr 16, 2021 at 12:56 PM Yonghong Song via lists.iovisor.org
<ys114321=gmail.com@...> wrote:

On Thu, Apr 15, 2021 at 11:09 PM Bradley Schatz
<bradley@...> wrote:

I still digging into this issue, and have hacked memleak/bcc to show addresses when they cant be resolved. In other places, I'm seeing jit complied java stack frames showing up alongside C ones, which is the expected behaviour when using Java with -XX:+PreserveFramePointer.

I'm still seeing these allocation traces with only two frames per the below.

11534336 bytes in 11 allocations from stack
[7efeebb115d4 unknown] [jna7632521838566054573.tmp ]
[7efec27f7440 unknown] [perf-15445.map]
14680064 bytes in 14 allocations from stack
[7efeebb115d4 unknown] [jna7632521838566054573.tmp ]
[7efec27f7380 unknown] [perf-15445.map]

I suspect the lower frame above is a libffi generated trampoline. Are there any known issues with the eBPF stack tracing infrastructure and such trampolines? Workarounds?
The bpf stack unwinder is using the kernel one which is the frame
pointer based and it may have issues with generated trampoline code
which may mess up frame pointer based stack chain. One possibility is
to use perf call-graph "dwarf" mode to get the raw data to user space
and use more powerful library like libunwind etc. to unwind the stack.
But I haven't do this together with bpf program yet, and cannot
describe whether and how to do "dwarf" call-graph with bpf program
together.
I briefly looked at the perf and kernel code and experimented with
perf dwarf mode.
It is possible for bpf to copy user stack to user space. You could do
the following:
- when you do perf event open, the sample_type needs to be set
properly. The following
is what callchain dwarf mode had:
sample_type = PERF_SAMPLE_IP|PERF_SAMPLE_TID|PERF_SAMPLE_TIME|
PERF_SAMPLE_ADDR|PERF_SAMPLE_CALLCHAIN|PERF_SAMPLE_CPU|
PERF_SAMPLE_PERIOD|PERF_SAMPLE_REGS_USER|
PERF_SAMPLE_STACK_USER|PERF_SAMPLE_DATA_SRC

I do not think you need all of them, you definitely need PERF_SAMPLE_REGS_USER,
maybe you can PERF_SAMPLE_RAW and PERF_SAMPLE_REGS_USER
to see whether it works or not.

In kernel header file linux:include/linux/perf_event.h, we have
struct perf_sample_data {
...
struct perf_regs regs_user;
struct perf_regs regs_intr;
u64 stack_user_size;
}

If you have access to perf_sample_data, you can read regs_user and find
stack pointer and then you can copy some bytes (say 2KB) to user space
for analysis. Using perf_event_output may be too expensive. perf is using
user mmap memory. You can also use a map for this purpose.

How to access perf_sample_data?
Typically sampling bpf program has the signature like
int do_sample(struct bpf_perf_event_data *ctx)
During program run, the program really got the following ctx
struct bpf_perf_event_data_kern {
bpf_user_pt_regs_t *regs;
struct perf_sample_data *data;
struct perf_event *event;
};
So you can use bpf_probe_read_kernel() to get 'data' and then other
information inside perf_sample_data.

You can experiment this way if you really like to see whether it can
solve your problem or not. If it works, I guess we can add a bpf helper
to do the copy as such copy length tends to be big.



Thanks,
Bradley



On 7/4/21, 9:04 pm, "Bradley Schatz" <bradley@...> wrote:

> What does '[unknown] [perf-18047.map]' mean? Does this mean
> perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
> symbolization won't be possible. Maybe you want to double check this?

The file perf-18047.map is there and from other parts of the stack trace I can see it being used to successfully resolve symbols.

Thanks!



On 7/4/21, 4:44 am, "Y Song" <ys114321@...> wrote:

On Mon, Apr 5, 2021 at 10:08 PM Bradley Schatz
<bradley@...> wrote:
>
> Thanks for the suggestion. I found a tunable to keep the JNI shared library in memory after loading. As you can see below, it is no longer showing as deleted.
>
> 13238272 bytes in 404 allocations from stack
> [unknown] [jna2576903844543447777.tmp]
> [unknown] [perf-18047.map]

I have no experience with perf-map-agent, but the following is what I guess:
[perf-18047.map] is used to find the mapping between address and symbol.
What does '[unknown] [perf-18047.map]' mean? Does this mean
perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
symbolization won't be possible. Maybe you want to double check this?

>
> No improvement in granularity though.
>
> In the VM I'm using -XX:+PreserveFramePointer -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints. In perf_maps_agent, I'm using "unfoldall"
>
> Any other suggestions?
>
> Thanks!
>
>
>
>
> On 3/4/21, 2:42 am, "Y Song" <ys114321@...> wrote:
>
> On Wed, Mar 31, 2021 at 11:25 PM Bradley Schatz
> <bradley@...> wrote:
> >
> > Hi,
> >
> >
> >
> > I’m just starting to come to grips with bcc & perf-map-agent for introspecting java on linux, with the goal of identifying what appears to be an off-heap memory leak (using memleak).
> >
> >
> >
> > I appear to be getting reliable stack decoding for jvm library code and for jit’ed java methods (see below for an example of the former). However I am seeing some very short stack traces which don’t seem to decode (the latter three stacks of below).
> >
> >
> >
> > It’s looking to me like the frame starting with “jna…” is likely the native JNI shared library for the FFI library “JNA”.
> >
> >
> >
> > Any suggestions as to why these latter three are so brief and/or how I can increase the resolution?
>
> I can see the file has been marked as deleted.
>
> 34603008 bytes in 33 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> 96468992 bytes in 92 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> So the file has been removed in userspace and current bcc won't be
> pass to parse it since it takes the file name as
> "jna9005484735610534564.tmp (deleted)"
> The file name is actually taken from /proc/<pid>/maps.
>
> I am not sure whether you can hack to parse "jna9005484735610534564.tmp" or not.
> But I would consider it is unsafe to do that as the original file
> related info may just
> exist in kernel and there is a reference to it. For user space, it is
> either gone or
> could be replaced by something else. So the safest way is to find a place to
> do symbolization before file is gone or keep tmp file a little bit longer.
> >
> >
> >
> > Apologies if this is the wrong place for such a question. Thank you for your help.
> >
> >
> >
> > Kind regards,
> >
> > Bradley
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 119408 bytes in 71 allocations from stack
> >
> > os::malloc(unsigned long, MemoryType, NativeCallStack const&)+0xb5 [libjvm.so]
> >
> > CodeBlob::set_oop_maps(OopMapSet*) [clone .part.5]+0x75 [libjvm.so]
> >
> > CodeBlob::CodeBlob(char const*, CodeBuffer*, int, int, int, int, OopMapSet*)+0xe3 [libjvm.so]
> >
> > nmethod::nmethod(Method*, int, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x4d [libjvm.so]
> >
> > nmethod::new_nmethod(methodHandle, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x219 [libjvm.so]
> >
> > ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool, RTMState)+0x1b1 [libjvm.so]
> >
> > Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0xe60 [libjvm.so]
> >
> > C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa3 [libjvm.so]
> >
> > CompileBroker::invoke_compiler_on_method(CompileTask*)+0x808 [libjvm.so]
> >
> > CompileBroker::compiler_thread_loop()+0x6d8 [libjvm.so]
> >
> > JavaThread::thread_main_inner()+0x1c7 [libjvm.so]
> >
> > JavaThread::run()+0x2fa [libjvm.so]
> >
> > java_start(Thread*)+0x102 [libjvm.so]
> >
> > start_thread+0xf3 [libpthread-2.28.so]
> >
> > 34603008 bytes in 33 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> > 96468992 bytes in 92 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> > 295698432 bytes in 282 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> >
> >
> >
>





Re: libffi trampolines and stack traces? : was Overly brief stack traces for Java/linux ?

Yonghong Song
 

On Thu, Apr 15, 2021 at 11:09 PM Bradley Schatz
<bradley@...> wrote:

I still digging into this issue, and have hacked memleak/bcc to show addresses when they cant be resolved. In other places, I'm seeing jit complied java stack frames showing up alongside C ones, which is the expected behaviour when using Java with -XX:+PreserveFramePointer.

I'm still seeing these allocation traces with only two frames per the below.

11534336 bytes in 11 allocations from stack
[7efeebb115d4 unknown] [jna7632521838566054573.tmp ]
[7efec27f7440 unknown] [perf-15445.map]
14680064 bytes in 14 allocations from stack
[7efeebb115d4 unknown] [jna7632521838566054573.tmp ]
[7efec27f7380 unknown] [perf-15445.map]

I suspect the lower frame above is a libffi generated trampoline. Are there any known issues with the eBPF stack tracing infrastructure and such trampolines? Workarounds?
The bpf stack unwinder is using the kernel one which is the frame
pointer based and it may have issues with generated trampoline code
which may mess up frame pointer based stack chain. One possibility is
to use perf call-graph "dwarf" mode to get the raw data to user space
and use more powerful library like libunwind etc. to unwind the stack.
But I haven't do this together with bpf program yet, and cannot
describe whether and how to do "dwarf" call-graph with bpf program
together.


Thanks,
Bradley



On 7/4/21, 9:04 pm, "Bradley Schatz" <bradley@...> wrote:

> What does '[unknown] [perf-18047.map]' mean? Does this mean
> perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
> symbolization won't be possible. Maybe you want to double check this?

The file perf-18047.map is there and from other parts of the stack trace I can see it being used to successfully resolve symbols.

Thanks!



On 7/4/21, 4:44 am, "Y Song" <ys114321@...> wrote:

On Mon, Apr 5, 2021 at 10:08 PM Bradley Schatz
<bradley@...> wrote:
>
> Thanks for the suggestion. I found a tunable to keep the JNI shared library in memory after loading. As you can see below, it is no longer showing as deleted.
>
> 13238272 bytes in 404 allocations from stack
> [unknown] [jna2576903844543447777.tmp]
> [unknown] [perf-18047.map]

I have no experience with perf-map-agent, but the following is what I guess:
[perf-18047.map] is used to find the mapping between address and symbol.
What does '[unknown] [perf-18047.map]' mean? Does this mean
perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
symbolization won't be possible. Maybe you want to double check this?

>
> No improvement in granularity though.
>
> In the VM I'm using -XX:+PreserveFramePointer -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints. In perf_maps_agent, I'm using "unfoldall"
>
> Any other suggestions?
>
> Thanks!
>
>
>
>
> On 3/4/21, 2:42 am, "Y Song" <ys114321@...> wrote:
>
> On Wed, Mar 31, 2021 at 11:25 PM Bradley Schatz
> <bradley@...> wrote:
> >
> > Hi,
> >
> >
> >
> > I’m just starting to come to grips with bcc & perf-map-agent for introspecting java on linux, with the goal of identifying what appears to be an off-heap memory leak (using memleak).
> >
> >
> >
> > I appear to be getting reliable stack decoding for jvm library code and for jit’ed java methods (see below for an example of the former). However I am seeing some very short stack traces which don’t seem to decode (the latter three stacks of below).
> >
> >
> >
> > It’s looking to me like the frame starting with “jna…” is likely the native JNI shared library for the FFI library “JNA”.
> >
> >
> >
> > Any suggestions as to why these latter three are so brief and/or how I can increase the resolution?
>
> I can see the file has been marked as deleted.
>
> 34603008 bytes in 33 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> 96468992 bytes in 92 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> So the file has been removed in userspace and current bcc won't be
> pass to parse it since it takes the file name as
> "jna9005484735610534564.tmp (deleted)"
> The file name is actually taken from /proc/<pid>/maps.
>
> I am not sure whether you can hack to parse "jna9005484735610534564.tmp" or not.
> But I would consider it is unsafe to do that as the original file
> related info may just
> exist in kernel and there is a reference to it. For user space, it is
> either gone or
> could be replaced by something else. So the safest way is to find a place to
> do symbolization before file is gone or keep tmp file a little bit longer.
> >
> >
> >
> > Apologies if this is the wrong place for such a question. Thank you for your help.
> >
> >
> >
> > Kind regards,
> >
> > Bradley
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 119408 bytes in 71 allocations from stack
> >
> > os::malloc(unsigned long, MemoryType, NativeCallStack const&)+0xb5 [libjvm.so]
> >
> > CodeBlob::set_oop_maps(OopMapSet*) [clone .part.5]+0x75 [libjvm.so]
> >
> > CodeBlob::CodeBlob(char const*, CodeBuffer*, int, int, int, int, OopMapSet*)+0xe3 [libjvm.so]
> >
> > nmethod::nmethod(Method*, int, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x4d [libjvm.so]
> >
> > nmethod::new_nmethod(methodHandle, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x219 [libjvm.so]
> >
> > ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool, RTMState)+0x1b1 [libjvm.so]
> >
> > Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0xe60 [libjvm.so]
> >
> > C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa3 [libjvm.so]
> >
> > CompileBroker::invoke_compiler_on_method(CompileTask*)+0x808 [libjvm.so]
> >
> > CompileBroker::compiler_thread_loop()+0x6d8 [libjvm.so]
> >
> > JavaThread::thread_main_inner()+0x1c7 [libjvm.so]
> >
> > JavaThread::run()+0x2fa [libjvm.so]
> >
> > java_start(Thread*)+0x102 [libjvm.so]
> >
> > start_thread+0xf3 [libpthread-2.28.so]
> >
> > 34603008 bytes in 33 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> > 96468992 bytes in 92 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> > 295698432 bytes in 282 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> >
> >
> >
>


libffi trampolines and stack traces? : was Overly brief stack traces for Java/linux ?

Bradley Schatz
 

I still digging into this issue, and have hacked memleak/bcc to show addresses when they cant be resolved. In other places, I'm seeing jit complied java stack frames showing up alongside C ones, which is the expected behaviour when using Java with -XX:+PreserveFramePointer.

I'm still seeing these allocation traces with only two frames per the below.

11534336 bytes in 11 allocations from stack
[7efeebb115d4 unknown] [jna7632521838566054573.tmp ]
[7efec27f7440 unknown] [perf-15445.map]
14680064 bytes in 14 allocations from stack
[7efeebb115d4 unknown] [jna7632521838566054573.tmp ]
[7efec27f7380 unknown] [perf-15445.map]

I suspect the lower frame above is a libffi generated trampoline. Are there any known issues with the eBPF stack tracing infrastructure and such trampolines? Workarounds?

Thanks,
Bradley



On 7/4/21, 9:04 pm, "Bradley Schatz" <bradley@...> wrote:

> What does '[unknown] [perf-18047.map]' mean? Does this mean
> perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
> symbolization won't be possible. Maybe you want to double check this?

The file perf-18047.map is there and from other parts of the stack trace I can see it being used to successfully resolve symbols.

Thanks!



On 7/4/21, 4:44 am, "Y Song" <ys114321@...> wrote:

On Mon, Apr 5, 2021 at 10:08 PM Bradley Schatz
<bradley@...> wrote:
>
> Thanks for the suggestion. I found a tunable to keep the JNI shared library in memory after loading. As you can see below, it is no longer showing as deleted.
>
> 13238272 bytes in 404 allocations from stack
> [unknown] [jna2576903844543447777.tmp]
> [unknown] [perf-18047.map]

I have no experience with perf-map-agent, but the following is what I guess:
[perf-18047.map] is used to find the mapping between address and symbol.
What does '[unknown] [perf-18047.map]' mean? Does this mean
perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
symbolization won't be possible. Maybe you want to double check this?

>
> No improvement in granularity though.
>
> In the VM I'm using -XX:+PreserveFramePointer -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints. In perf_maps_agent, I'm using "unfoldall"
>
> Any other suggestions?
>
> Thanks!
>
>
>
>
> On 3/4/21, 2:42 am, "Y Song" <ys114321@...> wrote:
>
> On Wed, Mar 31, 2021 at 11:25 PM Bradley Schatz
> <bradley@...> wrote:
> >
> > Hi,
> >
> >
> >
> > I’m just starting to come to grips with bcc & perf-map-agent for introspecting java on linux, with the goal of identifying what appears to be an off-heap memory leak (using memleak).
> >
> >
> >
> > I appear to be getting reliable stack decoding for jvm library code and for jit’ed java methods (see below for an example of the former). However I am seeing some very short stack traces which don’t seem to decode (the latter three stacks of below).
> >
> >
> >
> > It’s looking to me like the frame starting with “jna…” is likely the native JNI shared library for the FFI library “JNA”.
> >
> >
> >
> > Any suggestions as to why these latter three are so brief and/or how I can increase the resolution?
>
> I can see the file has been marked as deleted.
>
> 34603008 bytes in 33 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> 96468992 bytes in 92 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> So the file has been removed in userspace and current bcc won't be
> pass to parse it since it takes the file name as
> "jna9005484735610534564.tmp (deleted)"
> The file name is actually taken from /proc/<pid>/maps.
>
> I am not sure whether you can hack to parse "jna9005484735610534564.tmp" or not.
> But I would consider it is unsafe to do that as the original file
> related info may just
> exist in kernel and there is a reference to it. For user space, it is
> either gone or
> could be replaced by something else. So the safest way is to find a place to
> do symbolization before file is gone or keep tmp file a little bit longer.
> >
> >
> >
> > Apologies if this is the wrong place for such a question. Thank you for your help.
> >
> >
> >
> > Kind regards,
> >
> > Bradley
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 119408 bytes in 71 allocations from stack
> >
> > os::malloc(unsigned long, MemoryType, NativeCallStack const&)+0xb5 [libjvm.so]
> >
> > CodeBlob::set_oop_maps(OopMapSet*) [clone .part.5]+0x75 [libjvm.so]
> >
> > CodeBlob::CodeBlob(char const*, CodeBuffer*, int, int, int, int, OopMapSet*)+0xe3 [libjvm.so]
> >
> > nmethod::nmethod(Method*, int, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x4d [libjvm.so]
> >
> > nmethod::new_nmethod(methodHandle, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x219 [libjvm.so]
> >
> > ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool, RTMState)+0x1b1 [libjvm.so]
> >
> > Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0xe60 [libjvm.so]
> >
> > C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa3 [libjvm.so]
> >
> > CompileBroker::invoke_compiler_on_method(CompileTask*)+0x808 [libjvm.so]
> >
> > CompileBroker::compiler_thread_loop()+0x6d8 [libjvm.so]
> >
> > JavaThread::thread_main_inner()+0x1c7 [libjvm.so]
> >
> > JavaThread::run()+0x2fa [libjvm.so]
> >
> > java_start(Thread*)+0x102 [libjvm.so]
> >
> > start_thread+0xf3 [libpthread-2.28.so]
> >
> > 34603008 bytes in 33 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> > 96468992 bytes in 92 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> > 295698432 bytes in 282 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> >
> >
> >
>


Re: Overly brief stack traces for Java/linux ?

Bradley Schatz
 

What does '[unknown] [perf-18047.map]' mean? Does this mean
perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
symbolization won't be possible. Maybe you want to double check this?
The file perf-18047.map is there and from other parts of the stack trace I can see it being used to successfully resolve symbols.

Thanks!



On 7/4/21, 4:44 am, "Y Song" <ys114321@...> wrote:

On Mon, Apr 5, 2021 at 10:08 PM Bradley Schatz
<bradley@...> wrote:
>
> Thanks for the suggestion. I found a tunable to keep the JNI shared library in memory after loading. As you can see below, it is no longer showing as deleted.
>
> 13238272 bytes in 404 allocations from stack
> [unknown] [jna2576903844543447777.tmp]
> [unknown] [perf-18047.map]

I have no experience with perf-map-agent, but the following is what I guess:
[perf-18047.map] is used to find the mapping between address and symbol.
What does '[unknown] [perf-18047.map]' mean? Does this mean
perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
symbolization won't be possible. Maybe you want to double check this?

>
> No improvement in granularity though.
>
> In the VM I'm using -XX:+PreserveFramePointer -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints. In perf_maps_agent, I'm using "unfoldall"
>
> Any other suggestions?
>
> Thanks!
>
>
>
>
> On 3/4/21, 2:42 am, "Y Song" <ys114321@...> wrote:
>
> On Wed, Mar 31, 2021 at 11:25 PM Bradley Schatz
> <bradley@...> wrote:
> >
> > Hi,
> >
> >
> >
> > I’m just starting to come to grips with bcc & perf-map-agent for introspecting java on linux, with the goal of identifying what appears to be an off-heap memory leak (using memleak).
> >
> >
> >
> > I appear to be getting reliable stack decoding for jvm library code and for jit’ed java methods (see below for an example of the former). However I am seeing some very short stack traces which don’t seem to decode (the latter three stacks of below).
> >
> >
> >
> > It’s looking to me like the frame starting with “jna…” is likely the native JNI shared library for the FFI library “JNA”.
> >
> >
> >
> > Any suggestions as to why these latter three are so brief and/or how I can increase the resolution?
>
> I can see the file has been marked as deleted.
>
> 34603008 bytes in 33 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> 96468992 bytes in 92 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> So the file has been removed in userspace and current bcc won't be
> pass to parse it since it takes the file name as
> "jna9005484735610534564.tmp (deleted)"
> The file name is actually taken from /proc/<pid>/maps.
>
> I am not sure whether you can hack to parse "jna9005484735610534564.tmp" or not.
> But I would consider it is unsafe to do that as the original file
> related info may just
> exist in kernel and there is a reference to it. For user space, it is
> either gone or
> could be replaced by something else. So the safest way is to find a place to
> do symbolization before file is gone or keep tmp file a little bit longer.
> >
> >
> >
> > Apologies if this is the wrong place for such a question. Thank you for your help.
> >
> >
> >
> > Kind regards,
> >
> > Bradley
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > 119408 bytes in 71 allocations from stack
> >
> > os::malloc(unsigned long, MemoryType, NativeCallStack const&)+0xb5 [libjvm.so]
> >
> > CodeBlob::set_oop_maps(OopMapSet*) [clone .part.5]+0x75 [libjvm.so]
> >
> > CodeBlob::CodeBlob(char const*, CodeBuffer*, int, int, int, int, OopMapSet*)+0xe3 [libjvm.so]
> >
> > nmethod::nmethod(Method*, int, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x4d [libjvm.so]
> >
> > nmethod::new_nmethod(methodHandle, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x219 [libjvm.so]
> >
> > ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool, RTMState)+0x1b1 [libjvm.so]
> >
> > Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0xe60 [libjvm.so]
> >
> > C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa3 [libjvm.so]
> >
> > CompileBroker::invoke_compiler_on_method(CompileTask*)+0x808 [libjvm.so]
> >
> > CompileBroker::compiler_thread_loop()+0x6d8 [libjvm.so]
> >
> > JavaThread::thread_main_inner()+0x1c7 [libjvm.so]
> >
> > JavaThread::run()+0x2fa [libjvm.so]
> >
> > java_start(Thread*)+0x102 [libjvm.so]
> >
> > start_thread+0xf3 [libpthread-2.28.so]
> >
> > 34603008 bytes in 33 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> > 96468992 bytes in 92 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> > 295698432 bytes in 282 allocations from stack
> >
> > [unknown] [jna9005484735610534564.tmp (deleted)]
> >
> > [unknown] [perf-31566.map]
> >
> >
> >
> >
>


Re: Overly brief stack traces for Java/linux ?

Yonghong Song
 

On Mon, Apr 5, 2021 at 10:08 PM Bradley Schatz
<bradley@...> wrote:

Thanks for the suggestion. I found a tunable to keep the JNI shared library in memory after loading. As you can see below, it is no longer showing as deleted.

13238272 bytes in 404 allocations from stack
[unknown] [jna2576903844543447777.tmp]
[unknown] [perf-18047.map]
I have no experience with perf-map-agent, but the following is what I guess:
[perf-18047.map] is used to find the mapping between address and symbol.
What does '[unknown] [perf-18047.map]' mean? Does this mean
perf-18047.map is not found? If the perf-<pid>.map file cannot be found,
symbolization won't be possible. Maybe you want to double check this?


No improvement in granularity though.

In the VM I'm using -XX:+PreserveFramePointer -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints. In perf_maps_agent, I'm using "unfoldall"

Any other suggestions?

Thanks!




On 3/4/21, 2:42 am, "Y Song" <ys114321@...> wrote:

On Wed, Mar 31, 2021 at 11:25 PM Bradley Schatz
<bradley@...> wrote:
>
> Hi,
>
>
>
> I’m just starting to come to grips with bcc & perf-map-agent for introspecting java on linux, with the goal of identifying what appears to be an off-heap memory leak (using memleak).
>
>
>
> I appear to be getting reliable stack decoding for jvm library code and for jit’ed java methods (see below for an example of the former). However I am seeing some very short stack traces which don’t seem to decode (the latter three stacks of below).
>
>
>
> It’s looking to me like the frame starting with “jna…” is likely the native JNI shared library for the FFI library “JNA”.
>
>
>
> Any suggestions as to why these latter three are so brief and/or how I can increase the resolution?

I can see the file has been marked as deleted.

34603008 bytes in 33 allocations from stack

[unknown] [jna9005484735610534564.tmp (deleted)]

[unknown] [perf-31566.map]

96468992 bytes in 92 allocations from stack

[unknown] [jna9005484735610534564.tmp (deleted)]

[unknown] [perf-31566.map]

So the file has been removed in userspace and current bcc won't be
pass to parse it since it takes the file name as
"jna9005484735610534564.tmp (deleted)"
The file name is actually taken from /proc/<pid>/maps.

I am not sure whether you can hack to parse "jna9005484735610534564.tmp" or not.
But I would consider it is unsafe to do that as the original file
related info may just
exist in kernel and there is a reference to it. For user space, it is
either gone or
could be replaced by something else. So the safest way is to find a place to
do symbolization before file is gone or keep tmp file a little bit longer.
>
>
>
> Apologies if this is the wrong place for such a question. Thank you for your help.
>
>
>
> Kind regards,
>
> Bradley
>
>
>
>
>
>
>
>
>
> 119408 bytes in 71 allocations from stack
>
> os::malloc(unsigned long, MemoryType, NativeCallStack const&)+0xb5 [libjvm.so]
>
> CodeBlob::set_oop_maps(OopMapSet*) [clone .part.5]+0x75 [libjvm.so]
>
> CodeBlob::CodeBlob(char const*, CodeBuffer*, int, int, int, int, OopMapSet*)+0xe3 [libjvm.so]
>
> nmethod::nmethod(Method*, int, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x4d [libjvm.so]
>
> nmethod::new_nmethod(methodHandle, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x219 [libjvm.so]
>
> ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool, RTMState)+0x1b1 [libjvm.so]
>
> Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0xe60 [libjvm.so]
>
> C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa3 [libjvm.so]
>
> CompileBroker::invoke_compiler_on_method(CompileTask*)+0x808 [libjvm.so]
>
> CompileBroker::compiler_thread_loop()+0x6d8 [libjvm.so]
>
> JavaThread::thread_main_inner()+0x1c7 [libjvm.so]
>
> JavaThread::run()+0x2fa [libjvm.so]
>
> java_start(Thread*)+0x102 [libjvm.so]
>
> start_thread+0xf3 [libpthread-2.28.so]
>
> 34603008 bytes in 33 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> 96468992 bytes in 92 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> 295698432 bytes in 282 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
>
>
>


LPC 2021 Networking and BPF Track CFP

Daniel Borkmann
 

We are pleased to announce the Call for Proposals (CFP) for the Networking and
BPF track at the 2021 edition of the Linux Plumbers Conference (LPC), which is
planned to be held in Dublin, Ireland, on September 27th - 29th, 2021.

Note that if an in-person conference should prove to be impossible due to the
circumstances at that time, Linux Plumbers will switch to a virtual-only
conference. CFP submitters should ideally be able to give their presentation
in person, if circumstances permit, although presenting remotely will always
be possible.

This year's Networking and BPF track technical committee is comprised of:

David S. Miller <davem@...>
Jakub Kicinski <kuba@...>
Eric Dumazet <edumazet@...>
Alexei Starovoitov <ast@...>
Daniel Borkmann <daniel@...>
Andrii Nakryiko <andrii@...>

We are seeking proposals of 40 minutes in length (including Q&A discussion),
optionally accompanied by papers of 2 to 10 pages in length.

Any kind of advanced Linux networking and/or BPF related topic will be considered.

Please submit your proposals through the official LPC website at:

https://linuxplumbersconf.org/event/11/abstracts/

Make sure to select "Networking & BPF Summit" in the Track pull-down menu.

Proposals must be submitted by August 13th, and submitters will be notified of
acceptance by August 16th.

Final slides and papers (as PDF) are due on the first day of the conference.


Re: Overly brief stack traces for Java/linux ?

Bradley Schatz
 

Thanks for the suggestion. I found a tunable to keep the JNI shared library in memory after loading. As you can see below, it is no longer showing as deleted.

13238272 bytes in 404 allocations from stack
[unknown] [jna2576903844543447777.tmp]
[unknown] [perf-18047.map]

No improvement in granularity though.

In the VM I'm using -XX:+PreserveFramePointer -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints. In perf_maps_agent, I'm using "unfoldall"

Any other suggestions?

Thanks!




On 3/4/21, 2:42 am, "Y Song" <ys114321@...> wrote:

On Wed, Mar 31, 2021 at 11:25 PM Bradley Schatz
<bradley@...> wrote:
>
> Hi,
>
>
>
> I’m just starting to come to grips with bcc & perf-map-agent for introspecting java on linux, with the goal of identifying what appears to be an off-heap memory leak (using memleak).
>
>
>
> I appear to be getting reliable stack decoding for jvm library code and for jit’ed java methods (see below for an example of the former). However I am seeing some very short stack traces which don’t seem to decode (the latter three stacks of below).
>
>
>
> It’s looking to me like the frame starting with “jna…” is likely the native JNI shared library for the FFI library “JNA”.
>
>
>
> Any suggestions as to why these latter three are so brief and/or how I can increase the resolution?

I can see the file has been marked as deleted.

34603008 bytes in 33 allocations from stack

[unknown] [jna9005484735610534564.tmp (deleted)]

[unknown] [perf-31566.map]

96468992 bytes in 92 allocations from stack

[unknown] [jna9005484735610534564.tmp (deleted)]

[unknown] [perf-31566.map]

So the file has been removed in userspace and current bcc won't be
pass to parse it since it takes the file name as
"jna9005484735610534564.tmp (deleted)"
The file name is actually taken from /proc/<pid>/maps.

I am not sure whether you can hack to parse "jna9005484735610534564.tmp" or not.
But I would consider it is unsafe to do that as the original file
related info may just
exist in kernel and there is a reference to it. For user space, it is
either gone or
could be replaced by something else. So the safest way is to find a place to
do symbolization before file is gone or keep tmp file a little bit longer.
>
>
>
> Apologies if this is the wrong place for such a question. Thank you for your help.
>
>
>
> Kind regards,
>
> Bradley
>
>
>
>
>
>
>
>
>
> 119408 bytes in 71 allocations from stack
>
> os::malloc(unsigned long, MemoryType, NativeCallStack const&)+0xb5 [libjvm.so]
>
> CodeBlob::set_oop_maps(OopMapSet*) [clone .part.5]+0x75 [libjvm.so]
>
> CodeBlob::CodeBlob(char const*, CodeBuffer*, int, int, int, int, OopMapSet*)+0xe3 [libjvm.so]
>
> nmethod::nmethod(Method*, int, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x4d [libjvm.so]
>
> nmethod::new_nmethod(methodHandle, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x219 [libjvm.so]
>
> ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool, RTMState)+0x1b1 [libjvm.so]
>
> Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0xe60 [libjvm.so]
>
> C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa3 [libjvm.so]
>
> CompileBroker::invoke_compiler_on_method(CompileTask*)+0x808 [libjvm.so]
>
> CompileBroker::compiler_thread_loop()+0x6d8 [libjvm.so]
>
> JavaThread::thread_main_inner()+0x1c7 [libjvm.so]
>
> JavaThread::run()+0x2fa [libjvm.so]
>
> java_start(Thread*)+0x102 [libjvm.so]
>
> start_thread+0xf3 [libpthread-2.28.so]
>
> 34603008 bytes in 33 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> 96468992 bytes in 92 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
> 295698432 bytes in 282 allocations from stack
>
> [unknown] [jna9005484735610534564.tmp (deleted)]
>
> [unknown] [perf-31566.map]
>
>
>
>


Re: Overly brief stack traces for Java/linux ?

Yonghong Song
 

On Wed, Mar 31, 2021 at 11:25 PM Bradley Schatz
<bradley@...> wrote:

Hi,



I’m just starting to come to grips with bcc & perf-map-agent for introspecting java on linux, with the goal of identifying what appears to be an off-heap memory leak (using memleak).



I appear to be getting reliable stack decoding for jvm library code and for jit’ed java methods (see below for an example of the former). However I am seeing some very short stack traces which don’t seem to decode (the latter three stacks of below).



It’s looking to me like the frame starting with “jna…” is likely the native JNI shared library for the FFI library “JNA”.



Any suggestions as to why these latter three are so brief and/or how I can increase the resolution?
I can see the file has been marked as deleted.

34603008 bytes in 33 allocations from stack

[unknown] [jna9005484735610534564.tmp (deleted)]

[unknown] [perf-31566.map]

96468992 bytes in 92 allocations from stack

[unknown] [jna9005484735610534564.tmp (deleted)]

[unknown] [perf-31566.map]

So the file has been removed in userspace and current bcc won't be
pass to parse it since it takes the file name as
"jna9005484735610534564.tmp (deleted)"
The file name is actually taken from /proc/<pid>/maps.

I am not sure whether you can hack to parse "jna9005484735610534564.tmp" or not.
But I would consider it is unsafe to do that as the original file
related info may just
exist in kernel and there is a reference to it. For user space, it is
either gone or
could be replaced by something else. So the safest way is to find a place to
do symbolization before file is gone or keep tmp file a little bit longer.



Apologies if this is the wrong place for such a question. Thank you for your help.



Kind regards,

Bradley









119408 bytes in 71 allocations from stack

os::malloc(unsigned long, MemoryType, NativeCallStack const&)+0xb5 [libjvm.so]

CodeBlob::set_oop_maps(OopMapSet*) [clone .part.5]+0x75 [libjvm.so]

CodeBlob::CodeBlob(char const*, CodeBuffer*, int, int, int, int, OopMapSet*)+0xe3 [libjvm.so]

nmethod::nmethod(Method*, int, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x4d [libjvm.so]

nmethod::new_nmethod(methodHandle, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x219 [libjvm.so]

ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool, RTMState)+0x1b1 [libjvm.so]

Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0xe60 [libjvm.so]

C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa3 [libjvm.so]

CompileBroker::invoke_compiler_on_method(CompileTask*)+0x808 [libjvm.so]

CompileBroker::compiler_thread_loop()+0x6d8 [libjvm.so]

JavaThread::thread_main_inner()+0x1c7 [libjvm.so]

JavaThread::run()+0x2fa [libjvm.so]

java_start(Thread*)+0x102 [libjvm.so]

start_thread+0xf3 [libpthread-2.28.so]

34603008 bytes in 33 allocations from stack

[unknown] [jna9005484735610534564.tmp (deleted)]

[unknown] [perf-31566.map]

96468992 bytes in 92 allocations from stack

[unknown] [jna9005484735610534564.tmp (deleted)]

[unknown] [perf-31566.map]

295698432 bytes in 282 allocations from stack

[unknown] [jna9005484735610534564.tmp (deleted)]

[unknown] [perf-31566.map]




Overly brief stack traces for Java/linux ?

Bradley Schatz
 

Hi,

 

I’m just starting to come to grips with bcc & perf-map-agent for introspecting java on linux, with the goal of identifying what appears to be an off-heap memory leak (using memleak).

 

I appear to be getting reliable stack decoding for jvm library code and for jit’ed java methods (see below for an example of the former). However I am seeing some very short stack traces which don’t seem to decode (the latter three stacks of below).

 

It’s looking to me like the frame starting with “jna…” is likely the native JNI shared library for the FFI library “JNA”.

 

Any suggestions as to why these latter three are so brief and/or how I can increase the resolution?

 

Apologies if this is the wrong place for such a question. Thank you for your help.

 

Kind regards,

Bradley

 

 

 

 

       119408 bytes in 71 allocations from stack

              os::malloc(unsigned long, MemoryType, NativeCallStack const&)+0xb5 [libjvm.so]

              CodeBlob::set_oop_maps(OopMapSet*) [clone .part.5]+0x75 [libjvm.so]

              CodeBlob::CodeBlob(char const*, CodeBuffer*, int, int, int, int, OopMapSet*)+0xe3 [libjvm.so]

              nmethod::nmethod(Method*, int, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x4d [libjvm.so]

              nmethod::new_nmethod(methodHandle, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int)+0x219 [libjvm.so]

              ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, bool, bool, RTMState)+0x1b1 [libjvm.so]

              Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool)+0xe60 [libjvm.so]

              C2Compiler::compile_method(ciEnv*, ciMethod*, int)+0xa3 [libjvm.so]

              CompileBroker::invoke_compiler_on_method(CompileTask*)+0x808 [libjvm.so]

              CompileBroker::compiler_thread_loop()+0x6d8 [libjvm.so]

              JavaThread::thread_main_inner()+0x1c7 [libjvm.so]

              JavaThread::run()+0x2fa [libjvm.so]

              java_start(Thread*)+0x102 [libjvm.so]

              start_thread+0xf3 [libpthread-2.28.so]

       34603008 bytes in 33 allocations from stack

              [unknown] [jna9005484735610534564.tmp (deleted)]

              [unknown] [perf-31566.map]

       96468992 bytes in 92 allocations from stack

              [unknown] [jna9005484735610534564.tmp (deleted)]

              [unknown] [perf-31566.map]

       295698432 bytes in 282 allocations from stack

              [unknown] [jna9005484735610534564.tmp (deleted)]

              [unknown] [perf-31566.map]

 


Re: Questions about runqlen

Abel Wu
 

Hi Y Song,

On 3/21/21 1:38 AM, Y Song wrote:
On Tue, Mar 16, 2021 at 4:00 AM Abel Wu <wuyun.abel@...> wrote:

Hi, when I looked into the runqlen script yesterday, I found that,
sadly, I misunderstood the "queue length" all the time not only the
"length" part but also the "queue" part.
Could you file an "issue" for the question? This issue, the
questions/answers can be easily tracked.


Queue
=====
Only CFS runqueues are taken into account. This makes sense when
main workloads are all under CFS scheduler, which is common in
cloud scenario. But what I don't quite follow is that the selected
queue is task->se.cfs_rq which is from a task view, rather than the
top level cfs_rq from a cpu view. I suppose the task view is not
enough to draw the whole picture of saturation?

Length
======
Within this scope length means the number of schedulable entities,
that is cfs_rq->nr_running. From time sharing point of view, it is
OK because it represents how many units involved in scheduling of
this cfs_rq. But what about from execution point of view in which
the number of tasks (cfs_rq->h_nr_running) will be used?

And besides the above, without the shares information of each entity,
how could runqlen help us optimizing the performance? Maybe we should
always focus on occupancy rather than length?
There are some answers in this issue:
https://github.com/iovisor/bcc/issues/3093
To be accurate for cgroup/task-group environments, you may
need to traverse to the root. Could you check and experiment
whether this can solve your issue? if this is the case, we may
need to enhance runqlen.py. Maybe you could help provide
a pull request? Thanks!
Loop is forbidden in BPF programs (although bounded loop is
supported from linux-5.3, tracking down to NULL se->parent is
un-bounded). Maybe it's worth trying to get the definition of
struct rq? I will PR if made some progress.

Thanks,
Abel


It would be very much appreciated if someone can shed some light.

Thanks & Best regards,
Abel




Re: Questions about runqlen

Yonghong Song
 

On Tue, Mar 16, 2021 at 4:00 AM Abel Wu <wuyun.abel@...> wrote:

Hi, when I looked into the runqlen script yesterday, I found that,
sadly, I misunderstood the "queue length" all the time not only the
"length" part but also the "queue" part.
Could you file an "issue" for the question? This issue, the
questions/answers can be easily tracked.


Queue
=====
Only CFS runqueues are taken into account. This makes sense when
main workloads are all under CFS scheduler, which is common in
cloud scenario. But what I don't quite follow is that the selected
queue is task->se.cfs_rq which is from a task view, rather than the
top level cfs_rq from a cpu view. I suppose the task view is not
enough to draw the whole picture of saturation?

Length
======
Within this scope length means the number of schedulable entities,
that is cfs_rq->nr_running. From time sharing point of view, it is
OK because it represents how many units involved in scheduling of
this cfs_rq. But what about from execution point of view in which
the number of tasks (cfs_rq->h_nr_running) will be used?

And besides the above, without the shares information of each entity,
how could runqlen help us optimizing the performance? Maybe we should
always focus on occupancy rather than length?
There are some answers in this issue:
https://github.com/iovisor/bcc/issues/3093

To be accurate for cgroup/task-group environments, you may
need to traverse to the root. Could you check and experiment
whether this can solve your issue? if this is the case, we may
need to enhance runqlen.py. Maybe you could help provide
a pull request? Thanks!


It would be very much appreciated if someone can shed some light.

Thanks & Best regards,
Abel




21 - 40 of 2014