Hi all.

Currently debugging a very strange behaviour with eBPF hash maps and was wondering if anyone else has run into a similar issue? I am using libbpf-rs with BPF CO-RE and my kernel version is 5.9.14.

My setup: I have a map with some compound key and I am updating it once from userspace using libbpf and once (later) from a BPF program, using the same key both times, but with different values.

Here's the weird part: Somehow both key,value pairs are being stored in the map, according to output from bpftool. Even more bizarre, the value provided from userspace is essentially a "ghost value" the entire time -- all map lookups fail until the map has been updated from a BPF program as described above.

To be clear, the weirdness is two-fold:

1. Lookup should not fail after updating the map the first time; and
2. The second value should be overwriting the first one.

After performing both updates, here is the output from bpftool showcasing the weird behaviour:

[{
        "key": {
            "id": 3069983010007500772,
            "device": 48
        },
        "value": 10
    },{
        "key": {
            "id": 3069983010007500772,
            "device": 48
        },
        "value": 40
    }

]

This behaviour also seems to be inconsistent between different maps and yet consistent between different runs. For some maps, I get the expected result and for others I get this weirdness instead.

Is this possibly a bug in the kernel? Any assistance would be greatly appreciated.

Regards,
William

On Fri, Jan 15, 2021 at 12:42 PM William Findlay
<williamfindlay@...> wrote:

Hi all.

Currently debugging a very strange behaviour with eBPF hash maps and was wondering if anyone else has run into a similar issue? I am using libbpf-rs with BPF CO-RE and my kernel version is 5.9.14.

My setup: I have a map with some compound key and I am updating it once from userspace using libbpf and once (later) from a BPF program, using the same key both times, but with different values.

Here's the weird part: Somehow both key,value pairs are being stored in the map, according to output from bpftool. Even more bizarre, the value provided from userspace is essentially a "ghost value" the entire time -- all map lookups fail until the map has been updated from a BPF program as described above.

To be clear, the weirdness is two-fold:

Lookup should not fail after updating the map the first time; and
The second value should be overwriting the first one.

After performing both updates, here is the output from bpftool showcasing the weird behaviour:

[{
"key": {
"id": 3069983010007500772,
"device": 48
},
"value": 10
},{
"key": {
"id": 3069983010007500772,
"device": 48
},
"value": 40
}

]

Does your key data structure have padding? Different padding values
will cause different actual keys.

If padding is not an issue in your case, could you construct a test
case (best if no rust involved) so we can take a deep look?
You can file a task to document this issue if you intend to send a test case.
Thanks!

This behaviour also seems to be inconsistent between different maps and yet consistent between different runs. For some maps, I get the expected result and for others I get this weirdness instead.

Is this possibly a bug in the kernel? Any assistance would be greatly appreciated.

Regards,
William

[...]