Hi,

I'm running test_maps under net-next/samples/bpf/, commit
601009780635. The code logic all make sense but I got the assertion
errors / coredump for some unknown reason under different compiler
optimization flags (-O0 and -O2). The test_hashmap_sanity() works fine
but fails at test_percpu_hashmap_sanity().

First run with normal build (which has -O2)
[root@vm-dev bpf]# ./test_maps
test_maps: samples/bpf/test_maps.c:137: test_percpu_hashmap_sanity:
Assertion `bpf_lookup_elem(map_fd, &key, value) == -1 &&
(*__errno_location ()) == 2' failed.

where line test_maps.c: 137 is
/* check that key=2 is not found */
assert(bpf_lookup_elem(map_fd, &key, value) == -1 && errno == ENOENT);

Running it with strace, everything looks correct (bpf() return -1 and
errno == ENOENT)
[root@vm-dev bpf]# strace ./test_maps
bpf(0x1, 0x7ffcafe7f290, 0x30) = -1 ENOENT (No such file or directory)
write(2, "test_maps: samples/bpf/test_maps"..., 162test_maps:
samples/bpf/test_maps.c:137: test_percpu_hashmap_sanity: Assertion
`bpf_lookup_elem(map_fd, &key, value) == -1 && (*__errno_location ())
== 2' failed.
) = 162

Then I twisted Makefile with "-O0" instead of -O2, then line 137
passes but it fails at line 161.
test_maps: samples/bpf/test_maps.c:161: test_percpu_hashmap_sanity:
Assertion `(expected_key_mask & next_key) == next_key' failed.

using gdb to debug:
Aborted (core dumped)
(gdb) f 4
#4 0x0000000000401330 in test_percpu_hashmap_sanity (task=0,
data=0x0) at samples/bpf/test_maps.c:161
161 assert((expected_key_mask & next_key) == next_key);
(gdb) p key
$1 = 140736693748448
(gdb) p next_key
$2 = 2

It seems that the value of the key is corrupted. Any suggestion for
how to debug this?

Thank you!
William

On Tue, Jun 21, 2016 at 11:48 AM, William Tu via iovisor-dev
<iovisor-dev@...> wrote:

Hi,

I'm running test_maps under net-next/samples/bpf/, commit
601009780635. The code logic all make sense but I got the assertion
errors / coredump for some unknown reason under different compiler
optimization flags (-O0 and -O2). The test_hashmap_sanity() works fine
but fails at test_percpu_hashmap_sanity().

First run with normal build (which has -O2)
[root@vm-dev bpf]# ./test_maps
test_maps: samples/bpf/test_maps.c:137: test_percpu_hashmap_sanity:
Assertion `bpf_lookup_elem(map_fd, &key, value) == -1 &&
(*__errno_location ()) == 2' failed.

hmm that's really odd.
I don't see any issues with net-next.
# ulimit -l unlimited
# ./test_maps
test_maps: OK

where line test_maps.c: 137 is
/* check that key=2 is not found */
assert(bpf_lookup_elem(map_fd, &key, value) == -1 && errno == ENOENT);

Running it with strace, everything looks correct (bpf() return -1 and
errno == ENOENT)
[root@vm-dev bpf]# strace ./test_maps
bpf(0x1, 0x7ffcafe7f290, 0x30) = -1 ENOENT (No such file or directory)
write(2, "test_maps: samples/bpf/test_maps"..., 162test_maps:
samples/bpf/test_maps.c:137: test_percpu_hashmap_sanity: Assertion
`bpf_lookup_elem(map_fd, &key, value) == -1 && (*__errno_location ())
== 2' failed.
) = 162

Then I twisted Makefile with "-O0" instead of -O2, then line 137
passes but it fails at line 161.
test_maps: samples/bpf/test_maps.c:161: test_percpu_hashmap_sanity:
Assertion `(expected_key_mask & next_key) == next_key' failed.

that's even more weird.
May be some compiler issue.
I've tested with gcc 5.2 and 6.1

using gdb to debug:
Aborted (core dumped)
(gdb) f 4
#4 0x0000000000401330 in test_percpu_hashmap_sanity (task=0,
data=0x0) at samples/bpf/test_maps.c:161
161 assert((expected_key_mask & next_key) == next_key);
(gdb) p key
$1 = 140736693748448
(gdb) p next_key
$2 = 2

It seems that the value of the key is corrupted. Any suggestion for
how to debug this?

yeah. that's definitely odd.
Other than looking at generated asm. no other ideas.

Daniel, Brenden, Jesper, did you ever see anything like it?

On Tue, 21 Jun 2016 18:54:53 -0700
Alexei Starovoitov <alexei.starovoitov@...> wrote:

On Tue, Jun 21, 2016 at 11:48 AM, William Tu via iovisor-dev
<iovisor-dev@...> wrote:

Hi,

I'm running test_maps under net-next/samples/bpf/, commit
601009780635. The code logic all make sense but I got the assertion
errors / coredump for some unknown reason under different compiler
optimization flags (-O0 and -O2). The test_hashmap_sanity() works fine
but fails at test_percpu_hashmap_sanity().

First run with normal build (which has -O2)
[root@vm-dev bpf]# ./test_maps
test_maps: samples/bpf/test_maps.c:137: test_percpu_hashmap_sanity:
Assertion `bpf_lookup_elem(map_fd, &key, value) == -1 &&
(*__errno_location ()) == 2' failed.

[...]

Strange I get:

$ sudo ./test_maps
failed to create per-cpu arraymap 'Operation not permitted'

strace says:

bpf(BPF_MAP_CREATE, {map_type=0x6 /* BPF_MAP_TYPE_??? */, key_size=4, value_size=8, max_entries=20000}, 48) = -1 EPERM (Operation not permitted)

that's even more weird.
May be some compiler issue.
I've tested with gcc 5.2 and 6.1

[...]

Compile host: Fedora 22
* LLVM version 3.7.1 (llc and clang same version)
* gcc version 5.3.1 20151207 (Red Hat 5.3.1-2) (GCC)

Ran on a Fedora 23 host, running Alexei's kernel tree on "xdp" branch,
samples/bpf from same git tree.

Daniel, Brenden, Jesper, did you ever see anything like it?

Not working for me actually... but different error, strange.

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer

On Wed, Jun 22, 2016 at 8:54 AM, Jesper Dangaard Brouer
<brouer@...> wrote:

On Tue, 21 Jun 2016 18:54:53 -0700
Alexei Starovoitov <alexei.starovoitov@...> wrote:

On Tue, Jun 21, 2016 at 11:48 AM, William Tu via iovisor-dev
<iovisor-dev@...> wrote:

Hi,

I'm running test_maps under net-next/samples/bpf/, commit
601009780635. The code logic all make sense but I got the assertion
errors / coredump for some unknown reason under different compiler
optimization flags (-O0 and -O2). The test_hashmap_sanity() works fine
but fails at test_percpu_hashmap_sanity().

First run with normal build (which has -O2)
[root@vm-dev bpf]# ./test_maps
test_maps: samples/bpf/test_maps.c:137: test_percpu_hashmap_sanity:
Assertion `bpf_lookup_elem(map_fd, &key, value) == -1 &&
(*__errno_location ()) == 2' failed.

[...]

Strange I get:

$ sudo ./test_maps
failed to create per-cpu arraymap 'Operation not permitted'

yes. that's expected. Make sure to do 'ulimit -l unlimited'

On Wed, 22 Jun 2016 08:59:14 -0700
Alexei Starovoitov <alexei.starovoitov@...> wrote:

On Wed, Jun 22, 2016 at 8:54 AM, Jesper Dangaard Brouer
<brouer@...> wrote:

On Tue, 21 Jun 2016 18:54:53 -0700
Alexei Starovoitov <alexei.starovoitov@...> wrote:

On Tue, Jun 21, 2016 at 11:48 AM, William Tu via iovisor-dev
<iovisor-dev@...> wrote:

Hi,

I'm running test_maps under net-next/samples/bpf/, commit
601009780635. The code logic all make sense but I got the assertion
errors / coredump for some unknown reason under different compiler
optimization flags (-O0 and -O2). The test_hashmap_sanity() works fine
but fails at test_percpu_hashmap_sanity().

First run with normal build (which has -O2)
[root@vm-dev bpf]# ./test_maps
test_maps: samples/bpf/test_maps.c:137: test_percpu_hashmap_sanity:
Assertion `bpf_lookup_elem(map_fd, &key, value) == -1 &&
(*__errno_location ()) == 2' failed.

[...]

Strange I get:

$ sudo ./test_maps
failed to create per-cpu arraymap 'Operation not permitted'

yes. that's expected. Make sure to do 'ulimit -l unlimited'

# ulimit -l unlimited
# ./test_maps
test_maps: OK

Then it works, thanks. Maybe we could provide some better feedback from
the test program, as this might be a common/expected behavior.

--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer

On Wed, Jun 22, 2016 at 9:52 AM, Jesper Dangaard Brouer
<brouer@...> wrote:

On Wed, 22 Jun 2016 08:59:14 -0700
Alexei Starovoitov <alexei.starovoitov@...> wrote:

On Wed, Jun 22, 2016 at 8:54 AM, Jesper Dangaard Brouer
<brouer@...> wrote:

On Tue, 21 Jun 2016 18:54:53 -0700
Alexei Starovoitov <alexei.starovoitov@...> wrote:

On Tue, Jun 21, 2016 at 11:48 AM, William Tu via iovisor-dev
<iovisor-dev@...> wrote:

Hi,

I'm running test_maps under net-next/samples/bpf/, commit
601009780635. The code logic all make sense but I got the assertion
errors / coredump for some unknown reason under different compiler
optimization flags (-O0 and -O2). The test_hashmap_sanity() works fine
but fails at test_percpu_hashmap_sanity().

First run with normal build (which has -O2)
[root@vm-dev bpf]# ./test_maps
test_maps: samples/bpf/test_maps.c:137: test_percpu_hashmap_sanity:
Assertion `bpf_lookup_elem(map_fd, &key, value) == -1 &&
(*__errno_location ()) == 2' failed.

[...]

Strange I get:

$ sudo ./test_maps
failed to create per-cpu arraymap 'Operation not permitted'

yes. that's expected. Make sure to do 'ulimit -l unlimited'

# ulimit -l unlimited
# ./test_maps
test_maps: OK

Then it works, thanks. Maybe we could provide some better feedback from
the test program, as this might be a common/expected behavior.

probably the same patch as William did for sockex[12] is needed for test_maps.
bcc does it automatically.

Thanks!
I patched the test_maps.c to maximize the locked memory, but the
problem still exists. Let me try Alexei's kernel tree on "xdp" branch
to see if it works or not.

My environment:
Fedora 23 in VM
kernel: net-next master branch commit 601009780
gcc (GCC) 5.3.1 20160406 (Red Hat 5.3.1-6)
LLVM version 3.7.0

On Wed, Jun 22, 2016 at 10:07 AM, Alexei Starovoitov
<alexei.starovoitov@...> wrote:

On Wed, Jun 22, 2016 at 9:52 AM, Jesper Dangaard Brouer
<brouer@...> wrote:

On Wed, 22 Jun 2016 08:59:14 -0700
Alexei Starovoitov <alexei.starovoitov@...> wrote:

On Wed, Jun 22, 2016 at 8:54 AM, Jesper Dangaard Brouer
<brouer@...> wrote:

On Tue, 21 Jun 2016 18:54:53 -0700
Alexei Starovoitov <alexei.starovoitov@...> wrote:

On Tue, Jun 21, 2016 at 11:48 AM, William Tu via iovisor-dev
<iovisor-dev@...> wrote:

Hi,

I'm running test_maps under net-next/samples/bpf/, commit
601009780635. The code logic all make sense but I got the assertion
errors / coredump for some unknown reason under different compiler
optimization flags (-O0 and -O2). The test_hashmap_sanity() works fine
but fails at test_percpu_hashmap_sanity().

First run with normal build (which has -O2)
[root@vm-dev bpf]# ./test_maps
test_maps: samples/bpf/test_maps.c:137: test_percpu_hashmap_sanity:
Assertion `bpf_lookup_elem(map_fd, &key, value) == -1 &&
(*__errno_location ()) == 2' failed.

[...]

Strange I get:

$ sudo ./test_maps
failed to create per-cpu arraymap 'Operation not permitted'

yes. that's expected. Make sure to do 'ulimit -l unlimited'

# ulimit -l unlimited
# ./test_maps
test_maps: OK

Then it works, thanks. Maybe we could provide some better feedback from
the test program, as this might be a common/expected behavior.

probably the same patch as William did for sockex[12] is needed for test_maps.
bcc does it automatically.