Topics

Plans for libbpf packaging for distros?

Jesper Dangaard Brouer
 

Hi Jakub, Laura and Jiri Olsa (Cc others),

Subj: iovisor-dev] minutes: IO Visor TSC/Dev Meeting
(To: iovisor-dev <iovisor-dev@...>)
On Wed, 31 Oct 2018 14:30:25 -0700 "Brenden Blanco" <bblanco@...> wrote:
Jakub:
* working on getting libbpf packaged separately and released by distros
* FB has external mirror github.com/libbpf/libbpf
I noticed from the iovisor-dev minutes that you have plans for
packaging libbpf from the kernel tree. And via that I noticed the
github repo https://github.com/libbpf/libbpf, created by Yonghong Song.

I'm uncertain if it makes sense to maintain this library outside the
kernel git tree?!?

To be honest, I have very little knowledge about building RPMs and
other packages formats. I just wanted to point out that RHEL and
Fedora is now shipping bpftool, which also part of kernel git tree.

(Now I need input from Jiri Olsa and Laura to correct below statements:)

AFAIK bpftool RPM-package[1] is part of the "Source Package"
kernel-tools, which AFAIK gets build directly from the distro kernel
git tree via kernel.spec file. This also happens for perf
RPM-package[2] see section "Source Package" also point to kernel-tools.

So, my question is, can we ship/package libbpf in the same way?


Notice, that an increasing number of tools are linking/using libbpf,
e.g. perf, bpftool, Suricata, (selftests and samples/bpf).


[1] https://fedora.pkgs.org/28/fedora-x86_64/bpftool-4.16.0-1.fc28.x86_64.rpm.html
[2] https://fedora.pkgs.org/29/fedora-x86_64/perf-4.18.10-300.fc29.x86_64.rpm.html
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer

Jesper Dangaard Brouer
 

On Thu, 8 Nov 2018 12:37:06 -0200
Arnaldo Carvalho de Melo <acme@...> wrote:
Em Thu, Nov 08, 2018 at 02:37:54PM +0100, Jesper Dangaard Brouer escreveu:
Hi Jakub, Laura and Jiri Olsa (Cc others),

Subj: iovisor-dev] minutes: IO Visor TSC/Dev Meeting
(To: iovisor-dev <iovisor-dev@...>)
On Wed, 31 Oct 2018 14:30:25 -0700 "Brenden Blanco" <bblanco@...> wrote:
Jakub:
* working on getting libbpf packaged separately and released by distros
* FB has external mirror github.com/libbpf/libbpf
I noticed from the iovisor-dev minutes that you have plans for
packaging libbpf from the kernel tree. And via that I noticed the
github repo https://github.com/libbpf/libbpf, created by Yonghong Song.

I'm uncertain if it makes sense to maintain this library outside the
kernel git tree?!?
To ease access to the latest perf sources, we'll be making available
detached tarballs:

[acme@jouet linux]$ make help | grep perf
perf-tar-src-pkg - Build perf-4.20.0-rc1.tar source tarball
perf-targz-src-pkg - Build perf-4.20.0-rc1.tar.gz source tarball
perf-tarbz2-src-pkg - Build perf-4.20.0-rc1.tar.bz2 source tarball
perf-tarxz-src-pkg - Build perf-4.20.0-rc1.tar.xz source tarball
[acme@jouet linux]$

After each kernel release, we started with 4.19:

https://www.kernel.org/pub/linux/kernel/tools/perf/
And you basically also ship a version of libbpf in this tarball:

$ tar tvf ~/download/perf-4.19.0.tar.xz | grep lib/bpf
drwxrwxr-x root/root 0 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/
-rw-rw-r-- root/root 37 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/.gitignore
-rw-rw-r-- root/root 69 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/Build
-rw-rw-r-- root/root 6457 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/Makefile
-rw-rw-r-- root/root 16456 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/bpf.c
-rw-rw-r-- root/root 4440 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/bpf.h
-rw-rw-r-- root/root 7897 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/btf.c
-rw-rw-r-- root/root 775 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/btf.h
-rw-rw-r-- root/root 56905 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/libbpf.c
-rw-rw-r-- root/root 10683 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/libbpf.h
-rw-rw-r-- root/root 2380 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/libbpf_errno.c
-rw-rw-r-- root/root 4483 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/nlattr.c
-rw-rw-r-- root/root 1825 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/nlattr.h
-rw-rw-r-- root/root 479 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/str_error.c
-rw-rw-r-- root/root 152 2018-10-22 16:39 perf-4.19.0/tools/lib/bpf/str_error.h

Which gets compiled to libbpf.a and statically linked with perf.

Development continues in the kernel git tree, of course, and there
you'll be able to, using those top level kernel perf-tar-* targets to
get the bleeding edge, while those tarballs on
https://www.kernel.org/pub/linux/kernel/tools/perf/
help people wanting to try the latest released with older kernels, or to
test one previous release with a more recent kernel, to rule out
problems with some specific perf version.

Konstantin, the kernel.org admin accepted my suggestion for such a
directory name so that we could, in the future, perhaps have the other
tools/ living libraries and tools to follow this model, i.e. we would
have:

https://www.kernel.org/pub/linux/kernel/tools/lib/bpf/

etc.

- Arnaldo

To be honest, I have very little knowledge about building RPMs and
other packages formats. I just wanted to point out that RHEL and
Fedora is now shipping bpftool, which also part of kernel git tree.

(Now I need input from Jiri Olsa and Laura to correct below statements:)

AFAIK bpftool RPM-package[1] is part of the "Source Package"
kernel-tools, which AFAIK gets build directly from the distro kernel
git tree via kernel.spec file. This also happens for perf
RPM-package[2] see section "Source Package" also point to kernel-tools.

So, my question is, can we ship/package libbpf in the same way?


Notice, that an increasing number of tools are linking/using libbpf,
e.g. perf, bpftool, Suricata, (selftests and samples/bpf).


[1] https://fedora.pkgs.org/28/fedora-x86_64/bpftool-4.16.0-1.fc28.x86_64.rpm.html
[2] https://fedora.pkgs.org/29/fedora-x86_64/perf-4.18.10-300.fc29.x86_64.rpm.html
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer

Jakub Kicinski
 

On Thu, 8 Nov 2018 14:37:54 +0100, Jesper Dangaard Brouer wrote:
Hi Jakub, Laura and Jiri Olsa (Cc others),

Subj: iovisor-dev] minutes: IO Visor TSC/Dev Meeting
(To: iovisor-dev <iovisor-dev@...>)
On Wed, 31 Oct 2018 14:30:25 -0700 "Brenden Blanco" <bblanco@...> wrote:
Jakub:
* working on getting libbpf packaged separately and released by distros
* FB has external mirror github.com/libbpf/libbpf
I noticed from the iovisor-dev minutes that you have plans for
packaging libbpf from the kernel tree. And via that I noticed the
github repo https://github.com/libbpf/libbpf, created by Yonghong Song.

I'm uncertain if it makes sense to maintain this library outside the
kernel git tree?!?
To my understanding it's useful in two ways:
- some less fortunate distros (Debian) reportedly need a kernel build
to package bpftool, and for libbpf the same would have to happen.
At least that's what I'm told. So separate repo helps there a lot;
- we actually use the separate git repo as a submodule in our projects
(https://github.com/Netronome/bpf-samples will migrate there really
soon, just finishing code review).

So for us the git submodule thing works quite well until distros
package libbpf :)

To be honest, I have very little knowledge about building RPMs and
other packages formats. I just wanted to point out that RHEL and
Fedora is now shipping bpftool, which also part of kernel git tree.

(Now I need input from Jiri Olsa and Laura to correct below statements:)

AFAIK bpftool RPM-package[1] is part of the "Source Package"
kernel-tools, which AFAIK gets build directly from the distro kernel
git tree via kernel.spec file. This also happens for perf
RPM-package[2] see section "Source Package" also point to kernel-tools.

So, my question is, can we ship/package libbpf in the same way?


Notice, that an increasing number of tools are linking/using libbpf,
e.g. perf, bpftool, Suricata, (selftests and samples/bpf).


[1] https://fedora.pkgs.org/28/fedora-x86_64/bpftool-4.16.0-1.fc28.x86_64.rpm.html
[2] https://fedora.pkgs.org/29/fedora-x86_64/perf-4.18.10-300.fc29.x86_64.rpm.html
We were planning to do the same thing for libbpf. Let me copy paste the
patch to the package:

Add libbpf to kernel tools development libs. This library contains
functionality for loading and managing eBPF programs.

Signed-off-by: David Beckett <david.beckett@...>
---
kernel-tools.spec | 11 +++++++++++
1 file changed, 11 insertions(+)

diff --git a/kernel-tools.spec b/kernel-tools.spec
index 44d29ba..cf2f7a0 100644
--- a/kernel-tools.spec
+++ b/kernel-tools.spec
@@ -261,6 +261,9 @@ popd
pushd tools/gpio/
make
popd
+pushd tools/lib/bpf
+make
+popd
pushd tools/bpf/bpftool
make
popd
@@ -347,6 +350,9 @@ popd
pushd tools/kvm/kvm_stat
make INSTALL_ROOT=%{buildroot} install-tools
popd
+pushd tools/lib/bpf
+make DESTDIR=%{buildroot} prefix=%{_prefix} install install_headers
+popd
pushd tools/bpf/bpftool
make DESTDIR=%{buildroot} prefix=%{_prefix} bash_compdir=%{_sysconfdir}/bash_completion.d/ mandir=%{_mandir} install doc-install
popd
@@ -420,8 +426,13 @@ popd

%files -n kernel-tools-libs-devel
%{_libdir}/libcpupower.so
+%{_libdir}/libbpf.a
+%{_libdir}/libbpf.so
%{_includedir}/cpufreq.h
%{_includedir}/cpuidle.h
+%{_includedir}/bpf/bpf.h
+%{_includedir}/bpf/btf.h
+%{_includedir}/bpf/libbpf.h

%files -n bpftool
%{_sbindir}/bpftool

Fairly trivial patch, but since we learnt about the separate repo we
are migrating our internal projects and tests to that, then Alexei
suggested we need to add proper versioning to libbpf, and when all that
is done we can come back to packaging.

Alexei Starovoitov
 

On Thu, Nov 08, 2018 at 11:51:22AM -0800, Laura Abbott wrote:
On 11/8/18 7:48 AM, Jakub Kicinski wrote:
On Thu, 8 Nov 2018 14:37:54 +0100, Jesper Dangaard Brouer wrote:
Hi Jakub, Laura and Jiri Olsa (Cc others),

Subj: iovisor-dev] minutes: IO Visor TSC/Dev Meeting
(To: iovisor-dev <iovisor-dev@...>)
On Wed, 31 Oct 2018 14:30:25 -0700 "Brenden Blanco" <bblanco@...> wrote:
Jakub:
* working on getting libbpf packaged separately and released by distros
* FB has external mirror github.com/libbpf/libbpf
I noticed from the iovisor-dev minutes that you have plans for
packaging libbpf from the kernel tree. And via that I noticed the
github repo https://github.com/libbpf/libbpf, created by Yonghong Song.

I'm uncertain if it makes sense to maintain this library outside the
kernel git tree?!?
To my understanding it's useful in two ways:
- some less fortunate distros (Debian) reportedly need a kernel build
to package bpftool, and for libbpf the same would have to happen.
At least that's what I'm told. So separate repo helps there a lot;
- we actually use the separate git repo as a submodule in our projects
(https://github.com/Netronome/bpf-samples will migrate there really
soon, just finishing code review).

So for us the git submodule thing works quite well until distros
package libbpf :)
To be honest, I have very little knowledge about building RPMs and
other packages formats. I just wanted to point out that RHEL and
Fedora is now shipping bpftool, which also part of kernel git tree.

(Now I need input from Jiri Olsa and Laura to correct below statements:)

AFAIK bpftool RPM-package[1] is part of the "Source Package"
kernel-tools, which AFAIK gets build directly from the distro kernel
git tree via kernel.spec file. This also happens for perf
RPM-package[2] see section "Source Package" also point to kernel-tools.

So, my question is, can we ship/package libbpf in the same way?


Notice, that an increasing number of tools are linking/using libbpf,
e.g. perf, bpftool, Suricata, (selftests and samples/bpf).


[1] https://fedora.pkgs.org/28/fedora-x86_64/bpftool-4.16.0-1.fc28.x86_64.rpm.html
[2] https://fedora.pkgs.org/29/fedora-x86_64/perf-4.18.10-300.fc29.x86_64.rpm.html
We were planning to do the same thing for libbpf. Let me copy paste the
patch to the package:

Add libbpf to kernel tools development libs. This library contains
functionality for loading and managing eBPF programs.

Signed-off-by: David Beckett <david.beckett@...>
---
kernel-tools.spec | 11 +++++++++++
1 file changed, 11 insertions(+)

diff --git a/kernel-tools.spec b/kernel-tools.spec
index 44d29ba..cf2f7a0 100644
--- a/kernel-tools.spec
+++ b/kernel-tools.spec
@@ -261,6 +261,9 @@ popd
pushd tools/gpio/
make
popd
+pushd tools/lib/bpf
+make
+popd
pushd tools/bpf/bpftool
make
popd
@@ -347,6 +350,9 @@ popd
pushd tools/kvm/kvm_stat
make INSTALL_ROOT=%{buildroot} install-tools
popd
+pushd tools/lib/bpf
+make DESTDIR=%{buildroot} prefix=%{_prefix} install install_headers
+popd
pushd tools/bpf/bpftool
make DESTDIR=%{buildroot} prefix=%{_prefix} bash_compdir=%{_sysconfdir}/bash_completion.d/ mandir=%{_mandir} install doc-install
popd
@@ -420,8 +426,13 @@ popd
%files -n kernel-tools-libs-devel
%{_libdir}/libcpupower.so
+%{_libdir}/libbpf.a
+%{_libdir}/libbpf.so
%{_includedir}/cpufreq.h
%{_includedir}/cpuidle.h
+%{_includedir}/bpf/bpf.h
+%{_includedir}/bpf/btf.h
+%{_includedir}/bpf/libbpf.h
%files -n bpftool
%{_sbindir}/bpftool

Fairly trivial patch, but since we learnt about the separate repo we
are migrating our internal projects and tests to that, then Alexei
suggested we need to add proper versioning to libbpf, and when all that
is done we can come back to packaging.
Yes, this looks almost exactly like what I would expect to package
libpf. Fedora split out kernel-tools from the main kernel package
to reduce build dependencies (if something in kernel-tools breaks
it was blocking the build of the main kernel). I'm in favor of
continuing to move userspace tools out of the main kernel tree
for ease of packaging.
I think we're all on the same page.
Just to reiterate few points:
The source of truth of libbpf is in the kernel tree.
github/libbpf is a mirror of kernel sources plus few headers that
neceesary for the build.
Currently we don't have the mirror automated,
but the plan is to pull from the kernel tree continuously.
.spec files can live in github too,
and packages can be built out of it.

When we were talking to distor folks they suggested to make sure
that libbpf.SO have versioned symbols from day one.
As far as I know that's the only thing blocking creation of
official libbpf package.

Jesper Dangaard Brouer
 

On Thu, 8 Nov 2018 07:48:39 -0800
Jakub Kicinski <jakub.kicinski@...> wrote:
- we actually use the separate git repo as a submodule in our projects
(https://github.com/Netronome/bpf-samples will migrate there really
soon, just finishing code review).

So for us the git submodule thing works quite well until distros
package libbpf :)
That is an interesting use case.

I myself is also guilty[1] of copying the libbpf code from the kernel
and placing it on github to make my bpf-sample code work. I might take
the same approach of a git submodule... until distros package libbpf.


Notice my own disclaimer:
[1] https://github.com/netoptimizer/prototype-kernel/tree/master/kernel/samples/bpf/tools
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer