Hello BPF community,

I am looking for a way to move a user space program's disk I/O scheduling related logic down to kernel space, and then have the new kernel logic communicate with the user space program to make better I/O scheduling decisions. The reason that the user space program itself has I/O scheduling logic is because it needs to prioritize certain read or write requests.

I started looking at eBPF for that purpose. After doing some research, I learned that eBPF is very good at kernel profiling and tracing, but I didn't find much information about modifying kernel functions / data-structure using eBPF.

I am wondering:

1. Instead of calling eBPF function before / after calling a kernel function and then returning back to that kernel function, is it possible for eBPF programs to totally replace a kernel function or module logic?

2. Is it possible for eBPF programs to tamper the parameter and return value of a kernel function, or eBPF program can only read kernel data-structure but can not modify them? (some search indicates that it can not few years ago, but I am not sure if it is changed recently)

Thank you!
Jiada

[Edited Message Follows]

Nice, thanks Song. I am actually looking to track it till it is closed, so might have to remove that tag when the socket goes to closed state.
And once I have the concurrent connections info, say in a map, I am using XDP to drop the connections after they reach a threshold

So also wanted to ask if there is any way I can read the concurrent connections in XDP since the kernel already keeps track of them at /proc/net/tcp*?
That would help me avoid placing another tracepoint to track the connection count.

Appreciate your help!

Thanks,
Kanthi

On Thu, Oct 1, 2020 at 11:26 AM Y Song <ys114321@...> wrote:

On Tue, Sep 29, 2020 at 4:14 AM Kanthi P <Pavuluri.kanthi@...> wrote:

Hi,

I am looking for tracking inbound connections on a system using tracepoints/kprobes.

I was checking "trace_inet_sock_set_state", with which we can track the state changes during connection establishment and closure. It seems straightforward to track total connections, but since we only want inbound, one way would be to look at what are the ip addresses/ports on which a node listens to and while tracking the state changes, I can see if the local address/port matches to the one this system listens on and based on that make a decision whether its an inbound connection or not. This looks a bit roundabout way for me, so thought of reaching for suggestions to do it simpler.

Another way is to store the socker address when TCP_SYN_RECV to TCP_ESTABLISHED state change happens and during closure we can check if it is for this socket, so we know its inbound connection. But this would make the map size grow too high as we have about 50k concurrent connections.

Can you suggest a better way to do this?

Maybe you can use sk_local_storage? You can attach a piece of
information to the socket during TCP_SYN_RECV and later on during
TCP_ESTABLISHED to check that data, and you can delete that data from
the socket if you do not need it any more,
all in bpf program.

Thanks,
Kanthi

Hi,

Observing established connection counter discrepancy as 20% (30-40 connections mismatch out of 200) in one day that builds to 30% by day-2 and so on.

This observation is with this code

if (args->newstate == TCP_ESTABLISHED) 

                 __sync_fetch_and_add(val, 1); 

       if (args->oldstate == TCP_ESTABLISHED)       

                 __sync_fetch_and_add(val, -1);  

 } 

There was a typo in my first message.

Regards,
Ragalahari

I have to create a test-environment (based on vagrant) the last couple of days and i've done this with ubuntu 20.04 as base image.

Is the repository https://github.com/iovisor/vagrant still active?
If yes i would create a PR to update this Repository.

Hi,

I am looking for tracking inbound connections on a system using tracepoints/kprobes.

I was checking "trace_inet_sock_set_state", with which we can track the state changes during connection establishment and closure. It seems straightforward to track total connections, but since we only want inbound, one way would be to look at what are the ip addresses/ports on which a node listens to and while tracking the state changes, I can see if the local address/port matches to the one this system listens on and based on that make a decision whether its an inbound connection or not. This looks a bit roundabout way for me, so thought of reaching for suggestions to do it simpler.

Another way is to store the socker address when TCP_SYN_RECV to TCP_ESTABLISHED state change happens and during closure we can check if it is for this socket, so we know its inbound connection. But this would make the map size grow too high as we have about 50k concurrent connections.

Can you suggest a better way to do this?

Thanks,
Kanthi

Hi,

Is it possible to load a BPF program at boot time?

What I'm trying to achieve is to trace every single call to a certain
function since the kernel starts, without missing anything.

More specifically, I'm trying to debug iommu_alloc failures by looking
at the stacktrace to find out which subsystem/driver allocated too
many IOMMU slots on a ppc64le system, which I do not have direct
access to.

I've considered writing a systemd unit file that loads a BPF program
before the sysinit target[1], but I'm not sure if that's early enough.
An alternative seems to be to use boot-time tracing with ftrace[2]
instead (which I end up doing), but it requires recompiling the kernel
inorder to add tracepoints to retrieve the function call arguments,
and there isn't an easy way to stop tracing to prevent the tracing
buffer overflows (I end up writing a systemd unit file that sets a
ftrace event trigger that turns off tracing).

Maybe there is a better way to do something like this?


Much thanks,
Shung-Hsi Yu

[1]: https://www.freedesktop.org/software/systemd/man/bootup.html
[2]: https://www.kernel.org/doc/html/latest/trace/boottime-trace.html

Interestingly enough adding just -g in my Makefile built the BPF programs and allowed the BTF section to be found and properly loaded. My BPF program was loaded and is running properly with my desired functionality. I am confused though as to why the -g flag fixed this problem. Which according to the clang man page:

-g Generate debug information.

Is BTF information considered debug information? Is that in general or in this case? Is the this unexpected behavior? Perhaps a bug of clangs non -g compiled binaries with BPF? It would seem to me that the BTF information should not be purged from a non -g binary. I am interested to hear your thought on this Andrii!

Again, thank you so much for your help. There is no way I would have figured that out on my own.

Ian

-g Generate debug information.

Hello,

Here is the libbpf Logs at all levels for the open snoop program when using the pinned option for a map. This was tested on Linux Kernel v5.4 with libbpf 0.0.9, 0.1.0, and the current version. All the results of the logs were the same so I have only posted a single copy of it here. Let me know what you think and what the next steps might be! I appreciate the help and am having a good time trying to piece this together.

libbpf: section(14) .rel.eh_frame, size 32, link 15, flags 0, type=9

libbpf: skip relo .rel.eh_frame(14) for section(13)

libbpf: section(15) .symtab, size 408, link 1, flags 0, type=2

libbpf: BTF is required, but is missing or corrupted.

Ian