I was thinking about a 0.9.3 release and I noticed we had

https://github.com/iovisor/bpftrace/milestone/6

but it appears neglected.

Any thoughts on punting those issues and cutting a release now anyways?

We've merged a bunch of useful things (BTF, signed ints, int casts, better errors,
etc) since 0.9.2.

And yes, I am volunteering to drive the release.

On Wed, Oct 02, 2019 at 07:43:31PM +0200, Jiri Olsa wrote:

hi,
we'd like to have bcc linked with libbpf instead of the
github submodule, initial change is discussed in here:
https://github.com/iovisor/bcc/pull/2535

In order to do that, we need to have access to uapi headers
compatible with libbpf rpm, bcc is attaching and using them
during compilation.

I added them in the fedora spec below (not submitted yet),
so libbpf would carry those headers.

Thoughts? thanks,

I think it may break a bunch of people who rely on bcc being a single library.
What is the main motiviation to use libbpf as a shared library in libbcc?

I think we can have both options. libbpf as git submodule and as shared.
In practice git submodule is so much simpler to use and a lot less headaches.

Could you elaborate more on this ? Unfortunately, I cannot find the comments you gave me yesterday. (i think I chose a private reply)

Cilium has an idea to accelerate packet forward performance using sockops & sockhash when using istio service mesh, the code is here. But this function is heavily coupling with cilium codebase so I would like to extract the sockhash part from cilium. I find a demo code https://github.com/zachidan/ebpf-sockops and try to see if it really improve the performance. My test case is from https://github.com/istio/tools/tree/master/perf/benchmark . In this case, I setup two pods, fortio client and fortio server. And generate packet from client using kubectl -n $NAMESPACE exec -it $client_pod  -- fortio load -c 1 -qps 10000 -t 30s -a -r 0.00005 -httpbufferkb=128 "http://$server_svc_ip:8080/echo?size=1024", the qps decrease sharply from 6k to 200+ when apply sockmap prog. If I enter into the server pod and test using fortio load -c 1 -qps 10000 -t 30s -a -r 0.00005 -httpbufferkb=128 "http://127.0.0.1:8080/echo?size=1024", the qps increase from 6k to 9k.
In addition, I also override the bpf_redir function which always return SK_PASS and not call msg_redirect_hash,

__section("sk_msg")
int bpf_redir_proxy(struct sk_msg_md *msg)
{
    if (1)
        return SK_PASS;
    ...
    ...
}

and the qps also about 200+, so I think it is because the call to bpf_redir_proxy is expensive so the qps descrease sharply?

I also enter into the fortio server container and running test using fortio load -c 1 -qps 10000 -t 30s -a -r 0.00005 -httpbufferkb=128 "http://172.16.2.70:8080/echo?size=1024", here the ip 172.16.2.70 is the server IP(local ip), the result shows also 200+ qps…In this case, packet from client wiil first be redirected to envoy proxy, then the envoy proxy will send the packet to the server with dst address 127.0.0.1:8080

Why would this happen? How should I debug this? Need your help

Thanks,
forrest chen

What is your llvm version? What is your cmake output? Looks like llvm
libraries are not really linked?

Before I started to make a small change to bcc, I thought that I ought to verify that 'master' would actually build -- I'm running on Amazon Linux 2 AMI (HVM), SSD Volume Type - ami-0b69ea66ff7391e80. I followed the instructions in INSTALL.md and the make actually fails:

Scanning dependencies of target bcc-lua

[ 38%] Building C object src/lua/CMakeFiles/bcc-lua.dir/src/main.c.o

[ 39%] Linking C executable bcc-lua

[ 39%] Built target bcc-lua

Scanning dependencies of target bps

[ 39%] Building C object introspection/CMakeFiles/bps.dir/bps.c.o

[ 39%] Linking C executable bps

[ 39%] Built target bps

Scanning dependencies of target FollyRequestContextSwitch

[ 40%] Building CXX object examples/cpp/CMakeFiles/FollyRequestContextSwitch.dir/FollyRequestContextSwitch.cc.o

[ 40%] Linking CXX executable FollyRequestContextSwitch

../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::BTypeVisitor::genParamIndirectAssign(clang::FunctionDecl*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const**)':

b_frontend_action.cc:(.text+0x1d55): undefined reference to `clang::Rewriter::getRewrittenText[abi:cxx11](clang::SourceRange) const'

../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::BTypeVisitor::genParamDirectAssign(clang::FunctionDecl*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char const**)':

b_frontend_action.cc:(.text+0x32d7): undefined reference to `clang::Rewriter::getRewrittenText[abi:cxx11](clang::SourceRange) const'

../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::BTypeVisitor::VisitBinaryOperator(clang::BinaryOperator*)':

b_frontend_action.cc:(.text+0x4353): undefined reference to `clang::Rewriter::getRewrittenText[abi:cxx11](clang::SourceRange) const'

../../src/cc/frontends/clang/libclang_frontend.a(b_frontend_action.cc.o): In function `ebpf::BTypeVisitor::VisitFunctionDecl(clang::FunctionDecl*)':


Has anyone else seen this? 

Thanks

Philip

On Tue, Sep 3, 2019 at 9:40 AM Yonghong Song <ys114321@...> wrote:

Hi, All,

Brenden is on vacation this week. So I am sending out the notice. Does
anybody have any agenda to discuss on our pre-schedule 30min meeting
on Sept 4, 11:00am - 11:30am? If you do, please reply to this email.

No agenda means tomorrow conf call is cancelled.
And that would be few months in a row.
I think we should stop doing this email roll call as well.
The conf call was useful for long time, but not any more.
We have two bpf conferences a year where topics can be discussed face to face.
All other time it's best to use bpf@... mailing list for
questions and patches.

Hi, All,

Brenden is on vacation this week. So I am sending out the notice. Does
anybody have any agenda to discuss on our pre-schedule 30min meeting
on Sept 4, 11:00am - 11:30am? If you do, please reply to this email.

BTW, I will attend LPC 2019 (https://www.linuxplumbersconf.org/). If
you happen to be there, we can meet to discuss anything you are
interested for bcc, bpf, etc.

Thanks!

=== IO Visor Dev/TSC Meeting ===

Every 2 weeks on Wednesday, from Wednesday, January 25, 2017, to no end date
11:00 am | Pacific Daylight Time (San Francisco, GMT-07:00) | 30 min

https://bluejeans.com/568677804/

On Tue, Aug 20, 2019 at 12:05 PM Arnaldo Carvalho de Melo
<arnaldo.melo@...> wrote:

Em Tue, Aug 20, 2019 at 11:22:50AM -0700, Yonghong Song escreveu:

On Tue, Aug 20, 2019 at 11:02 AM Matthew Ahrens <mahrens@...> wrote:

On Tue, Aug 20, 2019 at 10:47 AM Yonghong Song <ys114321@...> wrote:

Also we have per cpu counter to prevent when bpf program interrupted then another bpf to run for tracing programs.

I think that means that if an interrupt fires while the bpf program is run, the interrupt will run, but if the interrupt causes another tracing event to fire, the associated bpf program will not run (i.e. the event will be ignored / dropped). Is that right?

Yes. See
https://github.com/torvalds/linux/blob/master/kernel/trace/bpf_trace.c#L88-L97

Yonghong, I thought there would be some counter to at least let users
know that drop happened, has this ever surfaced? Or is there some way to
know about those drops that I'm missing?

You did not miss anything. Currently, there are no counters to count those
drops due to nmi or due to bpf program already running on that cpu.

There is effort by Daniel Xu to expose nhit/nmisses counters
from k/uprobe trace infra. Even kprobe is not a miss, bpf program may not
fire due to the above reasons.
https://lore.kernel.org/bpf/20190820214819.16154-1-dxu@dxuuu.xyz/T/#t

debugfs has k/uprobe_profile to count nhit/nmisses from k/uprobe trace infra.

We could add a counter into trace_event_call->event to count hit/miss. The hit
can also be counted by bpf program itself. The "miss" should be rare, and
most bpf programs e.g. in bcc are designed to tolerate occasional probe miss,
which should not affect much on the final aggregation results.

How strongly do you feel such a bpf prog hit/miss counter for tracing programs
is needed?

- Arnaldo

Thanks Matt and Yonghong.  Info on the preempt_disable()/preempt_enable() region is exactly what I was looking for.

On Tue, Aug 20, 2019 at 11:02 AM Matthew Ahrens <mahrens@...> wrote:

On Tue, Aug 20, 2019 at 10:47 AM Yonghong Song <ys114321@...> wrote:

On Tue, Aug 20, 2019 at 10:38 AM Matthew Ahrens <mahrens@...> wrote:

On Mon, Aug 19, 2019 at 4:30 PM Yonghong Song <ys114321@...> wrote:

On Mon, Aug 19, 2019 at 1:54 PM <brad.lewis@...> wrote:

Hi all,

I'm trying to verify that there are no concurrency issues with an approach I'm using cpu_id as a key to a HASH_MAP. My understanding is that bcc disables preemption but the details are fuzzy and I haven't been able to find anything in the code. Can anyone shed some light on this?

preemption is a kernel thing, bcc does not disable it. You need to
check kernel configuration CONFIG_PREEMPT in your host.

When running bpf code from a kprobe / kretprobe, does anything ensure that cpu_id doesn't change while the bpf is running (e.g. due to preemption)? Does anything ensure that no other bpf code runs on this CPU while this kprobe is running (e.g. due to an interrupt firing and hitting a different kprobe)? If either of those things can happen, it seems difficult to atomically increment an entry in a HASH_MAP (even when using the cpu_id as a key).

bpf program run is wrapped in preempt_disable()/preempt_enable()
region. Also we have per cpu counter to prevent when bpf program
interrupted then another bpf to run for tracing programs. Therefore,
for tracing programs, the situation you mentioned in the above won't
happen.

The only thing which can happen is networking and tracing program
share the same map, e.g.,
networking bpf program can be nmi interrupted and a bpf tracing
program (perf_event type) may be running.

Great, thanks Yonghong! And I get that this is part of the infrastructure that calls into the bpf code (e.g. perf_events), not bcc.

Also we have per cpu counter to prevent when bpf program interrupted then another bpf to run for tracing programs.

I think that means that if an interrupt fires while the bpf program is run, the interrupt will run, but if the interrupt causes another tracing event to fire, the associated bpf program will not run (i.e. the event will be ignored / dropped). Is that right?

Yes. See
https://github.com/torvalds/linux/blob/master/kernel/trace/bpf_trace.c#L88-L97

--matt

--matt

On Tue, Aug 20, 2019 at 10:38 AM Matthew Ahrens <mahrens@...> wrote:

On Mon, Aug 19, 2019 at 4:30 PM Yonghong Song <ys114321@...> wrote:

On Mon, Aug 19, 2019 at 1:54 PM <brad.lewis@...> wrote:

Hi all,

I'm trying to verify that there are no concurrency issues with an approach I'm using cpu_id as a key to a HASH_MAP. My understanding is that bcc disables preemption but the details are fuzzy and I haven't been able to find anything in the code. Can anyone shed some light on this?

preemption is a kernel thing, bcc does not disable it. You need to
check kernel configuration CONFIG_PREEMPT in your host.

When running bpf code from a kprobe / kretprobe, does anything ensure that cpu_id doesn't change while the bpf is running (e.g. due to preemption)? Does anything ensure that no other bpf code runs on this CPU while this kprobe is running (e.g. due to an interrupt firing and hitting a different kprobe)? If either of those things can happen, it seems difficult to atomically increment an entry in a HASH_MAP (even when using the cpu_id as a key).

bpf program run is wrapped in preempt_disable()/preempt_enable()
region. Also we have per cpu counter to prevent when bpf program
interrupted then another bpf to run for tracing programs. Therefore,
for tracing programs, the situation you mentioned in the above won't
happen.

The only thing which can happen is networking and tracing program
share the same map, e.g.,
networking bpf program can be nmi interrupted and a bpf tracing
program (perf_event type) may be running.

--matt

Hi All,

We have the bi-weekly phone conference scheduled for two days from now, does
anybody have a discussion topic to add to the agenda? As a reminder, we are
planning to hold the meeting only if agenda items are proposed.

Cheers,
Brenden

On Mon, Aug 19, 2019 at 1:54 PM <brad.lewis@...> wrote:

Hi all,

I'm trying to verify that there are no concurrency issues with an approach I'm using cpu_id as a key to a HASH_MAP. My understanding is that bcc disables preemption but the details are fuzzy and I haven't been able to find anything in the code. Can anyone shed some light on this?

preemption is a kernel thing, bcc does not disable it. You need to
check kernel configuration CONFIG_PREEMPT in your host.

Thank you,

Brad Lewis

Hi all, 

I'm trying to verify that there are no concurrency issues with an approach I'm using cpu_id as a key to a HASH_MAP.   My understanding is that bcc disables preemption but the details are fuzzy and I haven't been able to find anything in the code.  Can anyone shed some light on this? 

 

Thank you, 

Brad Lewis 

 

 

Hi All,

Since we haven't received any agenda suggestions for the meeting
today, please consider it canceled.

Thanks,
Brenden

Hi all,

I attach an XDP prog(SKB mode) to host net device(enp0s9) and the prog will redirect packet to veth device.
If there's only veth device, I can redirect packet into netns through veth device in host, but if the veth device is attached in a bridge device, the redirect will fail(bpf_redirect_map return XDP_REDIRECT but no packet found in netns).

My sample code is here
The network topo shows below
note1: that if a remove br1, I can redirect successfully. 
note2: currently our k8s network mode has bridge device to forword packet. If bridge will make xdp not work, I have to consider to eliminate the bridge.
note3: I have test this sample in both 4.18 and 5.0.0, both of them failed.