Topics

minutes: IO Visor TSC/Dev Meeting

Brenden Blanco
 

Hi all,

Thanks for joining the discussion today. Here are the notes; however, this was
a longer discussion and I'm sure I missed some things.

Cheers,
Brenden

=== Discussion ===

Yonghong:
* Some internal BTF work
* Compiler support for static variables
* Some compile-once-run-everywhere work
* Looking for help with issues regarding libbpf packaging/dependencies
* Issue to continue offline, not concluded on the call
* Issue related to function->function call in bcc and compiler optimizations
* Jiong offers to debug the codegen using 32 bit mode

Saeed:
* XDP driver statistics standardization
* All drivers run same entry point for xdp progs
* Why not account stats here?
* Even though xdp program can implement its own statistics
* Many drivers are already paying stats accounting cost
* Just remove unused stats from driver?
* Stats may be used in debugging, but FB for instance is guarding with
static key, wouldn't want extra stats on by default
* Allocating resources for tx queue/redirect?
* Is there a better way to allocate resources when it isn't known that a
program will need queues
* One approach is to attach dummy bpf program
* Resource allocation point when configuring devmap?
* Seems like a clean enough solution, doesn't solve all cases but moves the
ball forward
* BTF metadata structure registration
* Should be queryable from userspace, don't yet have an API for that
* Netlink vs syscall?
* No silver bullet for all use cases
* Hesitation for creating a new object to describe existing objects (bpf
progs, maps)
* BTF is metadata conceptually different from maps, progs
* ethtool? unlikely due to lack of code ownership
* For buffers, something like devlink is more appropriate
* For BTF, bpf() syscall works
* BTF for statistics description (ethtool replacement?)

Daniel:
* verification of static data is working, patches coming soon

=== Attendees ===
Brenden Blanco
Michael Savisko
Alexei Starovoitov
Daniel Borkmann
Jakub Kicinski
Neerav Parikh
Paul Chaignon
Saeed
Marco Leogrande
Jiong Wang
Andrii Nakryiko
Yonghong Song
William Tu
Joe Stringer
John
Maciej Fijalkowski
Martin Lau
Mauricio Vasquez
Piotr Raczynski
Quillian Rutherford

Saeed Mahameed
 

Hi Brenden,

I am sending this on behalf of Michael Savisko, he is having some
difficulties sending emails to the iovisor list.

Michael is working on real world use cases for XDP acceleration.
He would like to present and discuss his work and analysis on
accelerating Katran load balancer [1] via meta data offloads.
He will need 10 minutes and will share some slides, i hope we can push
this to tomorrow's meeting agenda.

Katran modified code is on Github:
https://github.com/savisko/katran/tree/xdp_off

[1] https://code.fb.com/open-source/open-sourcing-katran-a-scalable-network-load-balancer/
Thanks,
Saeed.

On Wed, Mar 20, 2019 at 3:32 PM Brenden Blanco <bblanco@...> wrote:

Hi all,

Thanks for joining the discussion today. Here are the notes; however, this was
a longer discussion and I'm sure I missed some things.

Cheers,
Brenden

=== Discussion ===

Yonghong:
* Some internal BTF work
* Compiler support for static variables
* Some compile-once-run-everywhere work
* Looking for help with issues regarding libbpf packaging/dependencies
* Issue to continue offline, not concluded on the call
* Issue related to function->function call in bcc and compiler optimizations
* Jiong offers to debug the codegen using 32 bit mode

Saeed:
* XDP driver statistics standardization
* All drivers run same entry point for xdp progs
* Why not account stats here?
* Even though xdp program can implement its own statistics
* Many drivers are already paying stats accounting cost
* Just remove unused stats from driver?
* Stats may be used in debugging, but FB for instance is guarding with
static key, wouldn't want extra stats on by default
* Allocating resources for tx queue/redirect?
* Is there a better way to allocate resources when it isn't known that a
program will need queues
* One approach is to attach dummy bpf program
* Resource allocation point when configuring devmap?
* Seems like a clean enough solution, doesn't solve all cases but moves the
ball forward
* BTF metadata structure registration
* Should be queryable from userspace, don't yet have an API for that
* Netlink vs syscall?
* No silver bullet for all use cases
* Hesitation for creating a new object to describe existing objects (bpf
progs, maps)
* BTF is metadata conceptually different from maps, progs
* ethtool? unlikely due to lack of code ownership
* For buffers, something like devlink is more appropriate
* For BTF, bpf() syscall works
* BTF for statistics description (ethtool replacement?)

Daniel:
* verification of static data is working, patches coming soon

=== Attendees ===
Brenden Blanco
Michael Savisko
Alexei Starovoitov
Daniel Borkmann
Jakub Kicinski
Neerav Parikh
Paul Chaignon
Saeed
Marco Leogrande
Jiong Wang
Andrii Nakryiko
Yonghong Song
William Tu
Joe Stringer
John
Maciej Fijalkowski
Martin Lau
Mauricio Vasquez
Piotr Raczynski
Quillian Rutherford


Brenden Blanco
 

On Tue, Apr 2, 2019 at 10:48 AM Saeed Mahameed
<saeedm@...> wrote:

Hi Brenden,

I am sending this on behalf of Michael Savisko, he is having some
difficulties sending emails to the iovisor list.
Sending to the iovisor-dev mailer is gated by a signup requirement, in
order to reduce spam. The signup process should be pretty painless, I
believe it just requires going through an email validation step:
https://lists.iovisor.org/g/iovisor-dev/join

Michael is working on real world use cases for XDP acceleration.
He would like to present and discuss his work and analysis on
accelerating Katran load balancer [1] via meta data offloads.
He will need 10 minutes and will share some slides, i hope we can push
this to tomorrow's meeting agenda.
Sounds good!

Katran modified code is on Github:
https://github.com/savisko/katran/tree/xdp_off

[1] https://code.fb.com/open-source/open-sourcing-katran-a-scalable-network-load-balancer/
Thanks,
Saeed.

On Wed, Mar 20, 2019 at 3:32 PM Brenden Blanco <bblanco@...> wrote:

Hi all,

Thanks for joining the discussion today. Here are the notes; however, this was
a longer discussion and I'm sure I missed some things.

Cheers,
Brenden

=== Discussion ===

Yonghong:
* Some internal BTF work
* Compiler support for static variables
* Some compile-once-run-everywhere work
* Looking for help with issues regarding libbpf packaging/dependencies
* Issue to continue offline, not concluded on the call
* Issue related to function->function call in bcc and compiler optimizations
* Jiong offers to debug the codegen using 32 bit mode

Saeed:
* XDP driver statistics standardization
* All drivers run same entry point for xdp progs
* Why not account stats here?
* Even though xdp program can implement its own statistics
* Many drivers are already paying stats accounting cost
* Just remove unused stats from driver?
* Stats may be used in debugging, but FB for instance is guarding with
static key, wouldn't want extra stats on by default
* Allocating resources for tx queue/redirect?
* Is there a better way to allocate resources when it isn't known that a
program will need queues
* One approach is to attach dummy bpf program
* Resource allocation point when configuring devmap?
* Seems like a clean enough solution, doesn't solve all cases but moves the
ball forward
* BTF metadata structure registration
* Should be queryable from userspace, don't yet have an API for that
* Netlink vs syscall?
* No silver bullet for all use cases
* Hesitation for creating a new object to describe existing objects (bpf
progs, maps)
* BTF is metadata conceptually different from maps, progs
* ethtool? unlikely due to lack of code ownership
* For buffers, something like devlink is more appropriate
* For BTF, bpf() syscall works
* BTF for statistics description (ethtool replacement?)

Daniel:
* verification of static data is working, patches coming soon

=== Attendees ===
Brenden Blanco
Michael Savisko
Alexei Starovoitov
Daniel Borkmann
Jakub Kicinski
Neerav Parikh
Paul Chaignon
Saeed
Marco Leogrande
Jiong Wang
Andrii Nakryiko
Yonghong Song
William Tu
Joe Stringer
John
Maciej Fijalkowski
Martin Lau
Mauricio Vasquez
Piotr Raczynski
Quillian Rutherford


Brenden Blanco
 

Hi All,

Thanks for the good discussion today! Below are my notes.

Thanks,
Brenden

=== Discussion ===

Michael:
* https://github.com/savisko/katran/tree/xdp_off
* Mellanox presentation on XDP + Katran
* offload tc XDP programs to hardware nic
* Example application: Katran from Facebook
* Control is implemented as a C++ library (example is open source)
* Katran DP already implemented in XDP
* Parsing + extract flow ID
* lookup key generation
* counter update
* packet modified to forward to other IP
* Accelerate marking of flows in hardware
* XDP metadata to pass mark field from hardware to xdp program
* struct xdp_md_mark {
__u32 mark)
};
* if (mark_ptr + 1 <= data)
markID = mark_ptr->mark;
* per-CPU XDP map to convert mark -> real flow information
mark == 0 implies new flow
* original XDP slow-path has get_packet_dst() to create LRU mapping
* modified version uses perf event output to notify acceleration helper to
install flow mark in hardware
* perf results:
100 flows: 40+% performance improvements
10k flows: 0-50% performance improvements depending on #rx queues used
Software: 25Mpps
Hardware: 37-39Mpps
* considering changing implementation to mark real server instead of flow
id, to reduce number of entries kept in L1 cache

Yonghong:
* Compile once run anywhere work continues
* bitfield handling bugs in IR/debuginfo

Daniel:
* Global support work continues
* BTF side patches submitted to bpf mailing list
* tests included

Jiong:
* 32 bit patch set
* test methodology improvements
* updated patches later in the week
* some concerns around shifts, to be addressed in later improvements

Andrii:
* BTF and compile-once work integration
to share prototype tool with Saeed

Brendan:
* Is there a tool to measure queue latency in qdisc->netdev layer?
* debian/ubuntu are packaging bpftrace
* except libbcc renamed to libbpf_cc
* some issues with mixing iovisor's libbcc and debian's

Jesper:
* Fedora adding packaging support for libbpf

Alexei:
* systemd also adding support for libbpf - link to be provided?

=== Attendees ===
Brenden Blanco
Andril Nakryiko
Daniel Borkmann
Jesper Brouer
Jiong Wang
Marco Leogrande
Michael Savisko
Paul Chaignon
Quentin Monnet
Alexei Starovoitov
Saeed
Flavio
Rony
Jonathan Lemon
Brendan Gregg
Dan Siemon
Joe Stringer
John

Yonghong Song
 

Hi, Jiong,

To follow up the iovisor meeting discussion, the below is my prototype
for an end_loop
instruction in llvm:
https://github.com/yonghong-song/llvm/commit/b83226772100317092cae6478229ed6ca3b9903c
The goal is to help verifier to just focus on these marked cases,
rejecting any other backedges.

Please let me and John know if you get some better idea for bounded
loop support in llvm/verifier.

Thanks,

Yonghong

On Wed, Apr 3, 2019 at 11:49 AM Brenden Blanco <bblanco@...> wrote:

Hi All,

Thanks for the good discussion today! Below are my notes.

Thanks,
Brenden

=== Discussion ===
[...]
Yonghong:
* Compile once run anywhere work continues
* bitfield handling bugs in IR/debuginfo

Daniel:
* Global support work continues
* BTF side patches submitted to bpf mailing list
* tests included

Jiong:
* 32 bit patch set
* test methodology improvements
* updated patches later in the week
* some concerns around shifts, to be addressed in later improvements

Andrii:
* BTF and compile-once work integration
to share prototype tool with Saeed

Brendan:
* Is there a tool to measure queue latency in qdisc->netdev layer?
* debian/ubuntu are packaging bpftrace
* except libbcc renamed to libbpf_cc
* some issues with mixing iovisor's libbcc and debian's

Jesper:
* Fedora adding packaging support for libbpf

Alexei:
* systemd also adding support for libbpf - link to be provided?

=== Attendees ===
Brenden Blanco
Andril Nakryiko
Daniel Borkmann
Jesper Brouer
Jiong Wang
Marco Leogrande
Michael Savisko
Paul Chaignon
Quentin Monnet
Alexei Starovoitov
Saeed
Flavio
Rony
Jonathan Lemon
Brendan Gregg
Dan Siemon
Joe Stringer
John


Jiong Wang
 

Great, will have a look, thanks!

Regards,
Jiong
On 05/04/2019 07:45, Y Song wrote:

Hi, Jiong,

To follow up the iovisor meeting discussion, the below is my prototype
for an end_loop
instruction in llvm:
https://github.com/yonghong-song/llvm/commit/b83226772100317092cae6478229ed6ca3b9903c
The goal is to help verifier to just focus on these marked cases,
rejecting any other backedges.

Please let me and John know if you get some better idea for bounded
loop support in llvm/verifier.

Thanks,

Yonghong

On Wed, Apr 3, 2019 at 11:49 AM Brenden Blanco <bblanco@...> wrote:
Hi All,

Thanks for the good discussion today! Below are my notes.

Thanks,
Brenden

=== Discussion ===
[...]
Yonghong:
* Compile once run anywhere work continues
  * bitfield handling bugs in IR/debuginfo

Daniel:
* Global support work continues
  * BTF side patches submitted to bpf mailing list
  * tests included

Jiong:
* 32 bit patch set
  * test methodology improvements
  * updated patches later in the week
  * some concerns around shifts, to be addressed in later improvements

Andrii:
* BTF and compile-once work integration
  to share prototype tool with Saeed

Brendan:
* Is there a tool to measure queue latency in qdisc->netdev layer?
  * debian/ubuntu are packaging bpftrace
  * except libbcc renamed to libbpf_cc
  * some issues with mixing iovisor's libbcc and debian's

Jesper:
* Fedora adding packaging support for libbpf

Alexei:
* systemd also adding support for libbpf - link to be provided?

=== Attendees ===
Brenden Blanco
Andril Nakryiko
Daniel Borkmann
Jesper Brouer
Jiong Wang
Marco Leogrande
Michael Savisko
Paul Chaignon
Quentin Monnet
Alexei Starovoitov
Saeed
Flavio
Rony
Jonathan Lemon
Brendan Gregg
Dan Siemon
Joe Stringer
John