Tracing malloc/free calls in a Kubernetes Pod


Lorenzo Fontana
 


On Sun, 14 Jun 2020 at 20:32 <adelstaging+iovisor@...> wrote:
Hey folks,

I have been experimenting with bpf(trace) on a Kubernetes cluster and have gotten kubectl-trace instrumenting an application running in a Pod. Now I want to instrument the code to chase down a memory leak happening in one of the applications - originally I was hoping to use the memleak BCC tool but it seemed a pain to get it working generically, so I turned my attention to bpftrace and kubectl-trace. The problem I'm running into is I believe I need to instrument libc to listen on those calls, but I don't know of a way to point at the Pod's libc in kubectl-trace.

As I understand it, much of kubectl-trace's functionality is figuring out a Pod's process ID in the node's root namespace and exposing it via $container_pid, but the bpftrace program itself still just runs on the node, which makes sense. With the $container_pid variable we can then point at an application process via the node's procfs, i.e. /proc/$container_pid/exe. However I have not been able to figure out how to point to the $container_pid's libc, if that is at all possible?

Any suggestions would be much appreciated. Thanks!
_._,_._,_

Links:

You receive all messages sent to this group.

View/Reply Online (#1863) | Reply To Sender | Reply To Group | Mute This Topic | New Topic

Your Subscription | Contact Group Owner | Unsubscribe [fontanalorenz@...]



Replying here again for the record since you posted the same question on the k8s slack.

Kubectl trace replaces $container_pid so you can access the pid folder in the host proc. it’s not specific only for exe.
That means that you can instrument anything from that directory using the root symlink inside that pid folder.

E.g: /proc/$container_pid/root/lib/yourlib.so


Thanks for the PR today,
Lore