Weird behaviour when updating a hash map from userspace
Hi all.
Currently debugging a very strange behaviour with eBPF hash maps and was wondering if anyone else has run into a similar issue? I am using libbpf-rs with BPF CO-RE and my kernel version is 5.9.14.
My setup: I have a map with some compound key and I am updating it once from userspace using libbpf and once (later) from a BPF program, using the same key both times, but with different values.
Here's the weird part: Somehow both key,value pairs are being stored in the map, according to output from bpftool. Even more bizarre, the value provided from userspace is essentially a "ghost value" the entire time -- all map lookups fail until the map has been updated from a BPF program as described above.
To be clear, the weirdness is two-fold:
- Lookup should not fail after updating the map the first time; and
- The second value should be overwriting the first one.
After performing both updates, here is the output from bpftool showcasing the weird behaviour:
[{"key": {
"id": 3069983010007500772,
"device": 48
},
"value": 10
},{
"key": {
"id": 3069983010007500772,
"device": 48
},
"value": 40
}
]
This behaviour also seems to be inconsistent between different maps and yet consistent between different runs. For some maps, I get the expected result and for others I get this weirdness instead.
Is this possibly a bug in the kernel? Any assistance would be greatly appreciated.
Regards,
William
<williamfindlay@...> wrote:
Does your key data structure have padding? Different padding values
Hi all.
Currently debugging a very strange behaviour with eBPF hash maps and was wondering if anyone else has run into a similar issue? I am using libbpf-rs with BPF CO-RE and my kernel version is 5.9.14.
My setup: I have a map with some compound key and I am updating it once from userspace using libbpf and once (later) from a BPF program, using the same key both times, but with different values.
Here's the weird part: Somehow both key,value pairs are being stored in the map, according to output from bpftool. Even more bizarre, the value provided from userspace is essentially a "ghost value" the entire time -- all map lookups fail until the map has been updated from a BPF program as described above.
To be clear, the weirdness is two-fold:
Lookup should not fail after updating the map the first time; and
The second value should be overwriting the first one.
After performing both updates, here is the output from bpftool showcasing the weird behaviour:
[{
"key": {
"id": 3069983010007500772,
"device": 48
},
"value": 10
},{
"key": {
"id": 3069983010007500772,
"device": 48
},
"value": 40
}
]
will cause different actual keys.
If padding is not an issue in your case, could you construct a test
case (best if no rust involved) so we can take a deep look?
You can file a task to document this issue if you intend to send a test case.
Thanks!
[...]
This behaviour also seems to be inconsistent between different maps and yet consistent between different runs. For some maps, I get the expected result and for others I get this weirdness instead.
Is this possibly a bug in the kernel? Any assistance would be greatly appreciated.
Regards,
William