Hello folks,

We have an eBPF program that shares a 64-bits integer with userspace using an array map. The eBPF program only reads the integer while the user-space application only writes it.

We know this is possible that under some conditions, for example the integer is not aligned to cache line size in x86_64, the reader can see a partially updated value. We wonder if eBPF have some guarantees about atomicity of the load operation in this case, or if this is totally dependent on the target architecture.

Is there a way to implement this without using bpf_spin_lock?

Thanks,

Mauricio.

On Thu, Feb 7, 2019 at 8:15 AM Mauricio Vasquez
<mauricio.vasquez@...> wrote:

Hello folks,

We have an eBPF program that shares a 64-bits integer with userspace
using an array map. The eBPF program only reads the integer while the
user-space application only writes it.

We know this is possible that under some conditions, for example the
integer is not aligned to cache line size in x86_64, the reader can see
a partially updated value. We wonder if eBPF have some guarantees about

not just cache line, you mean not in its naturally aligned boundary?
One load/store is actually broken into multiple loads/stores by the compiler?

atomicity of the load operation in this case, or if this is totally
dependent on the target architecture.

Is there a way to implement this without using bpf_spin_lock?

So you mean both user space and bpf program are using bpf_spin_lock?

Thanks,

Mauricio.

On 2/7/19 3:49 PM, Yonghong Song wrote:

On Thu, Feb 7, 2019 at 8:15 AM Mauricio Vasquez
<mauricio.vasquez@...> wrote:

Hello folks,

We have an eBPF program that shares a 64-bits integer with userspace
using an array map. The eBPF program only reads the integer while the
user-space application only writes it.

We know this is possible that under some conditions, for example the
integer is not aligned to cache line size in x86_64, the reader can see
a partially updated value. We wonder if eBPF have some guarantees about

not just cache line, you mean not in its naturally aligned boundary?

Exactly.

One load/store is actually broken into multiple loads/stores by the compiler?

atomicity of the load operation in this case, or if this is totally
dependent on the target architecture.

Is there a way to implement this without using bpf_spin_lock?

So you mean both user space and bpf program are using bpf_spin_lock?

Please consider this example based on BCC https://gist.github.com/mauriciovasquezbernal/73ade7609eadcf23fb254401ed945cb3. In this case the counter member is naturally aligned, so in x86_64 the program will never see a partial update. However this could be not true for all architectures. AFAIK bpf_spin_lock is the only way to implement this example in a safe way for all architectures.

In other words, it is not possible to make assumptions about load or store operations atomicity in eBPF, this is dependent on the architecture where the program is run.

Is that statement correct?

Thanks,

Mauricio.

Thanks,

Mauricio.

On Fri, Feb 8, 2019 at 2:48 PM Mauricio Vasquez
<mauricio.vasquez@...> wrote:

On 2/7/19 3:49 PM, Yonghong Song wrote:

On Thu, Feb 7, 2019 at 8:15 AM Mauricio Vasquez
<mauricio.vasquez@...> wrote:

Hello folks,

We have an eBPF program that shares a 64-bits integer with userspace
using an array map. The eBPF program only reads the integer while the
user-space application only writes it.

We know this is possible that under some conditions, for example the
integer is not aligned to cache line size in x86_64, the reader can see
a partially updated value. We wonder if eBPF have some guarantees about

not just cache line, you mean not in its naturally aligned boundary?

Exactly.

One load/store is actually broken into multiple loads/stores by the compiler?

atomicity of the load operation in this case, or if this is totally
dependent on the target architecture.

Is there a way to implement this without using bpf_spin_lock?

So you mean both user space and bpf program are using bpf_spin_lock?

Please consider this example based on BCC
https://gist.github.com/mauriciovasquezbernal/73ade7609eadcf23fb254401ed945cb3.
In this case the counter member is naturally aligned, so in x86_64 the
program will never see a partial update. However this could be not true
for all architectures. AFAIK bpf_spin_lock is the only way to implement
this example in a safe way for all architectures.

In other words, it is not possible to make assumptions about load or
store operations atomicity in eBPF, this is dependent on the
architecture where the program is run.

Is that statement correct?

There are some memory model discussion in the netdev mailing list.
I think this shows bpf memory model is really tied to underlying arch indeed.

Some kinds of lock is the only way to guarantee the atomacity.
bpf_spin_lock is a kernel bpf helper. I guess user space could implement
the spin_lock for map element as well based on how kernel bpf_spin_lock
is implemented.

Thanks,

Mauricio.

Thanks,

Mauricio.

On 2/9/19 6:31 PM, Yonghong Song wrote:

On Fri, Feb 8, 2019 at 2:48 PM Mauricio Vasquez
<mauricio.vasquez@...> wrote:

On 2/7/19 3:49 PM, Yonghong Song wrote:

On Thu, Feb 7, 2019 at 8:15 AM Mauricio Vasquez
<mauricio.vasquez@...> wrote:

Hello folks,

We have an eBPF program that shares a 64-bits integer with userspace
using an array map. The eBPF program only reads the integer while the
user-space application only writes it.

We know this is possible that under some conditions, for example the
integer is not aligned to cache line size in x86_64, the reader can see
a partially updated value. We wonder if eBPF have some guarantees about

not just cache line, you mean not in its naturally aligned boundary?

Exactly.

One load/store is actually broken into multiple loads/stores by the compiler?

atomicity of the load operation in this case, or if this is totally
dependent on the target architecture.

Is there a way to implement this without using bpf_spin_lock?

So you mean both user space and bpf program are using bpf_spin_lock?

Please consider this example based on BCC
https://gist.github.com/mauriciovasquezbernal/73ade7609eadcf23fb254401ed945cb3.
In this case the counter member is naturally aligned, so in x86_64 the
program will never see a partial update. However this could be not true
for all architectures. AFAIK bpf_spin_lock is the only way to implement
this example in a safe way for all architectures.

In other words, it is not possible to make assumptions about load or
store operations atomicity in eBPF, this is dependent on the
architecture where the program is run.

Is that statement correct?

There are some memory model discussion in the netdev mailing list.
I think this shows bpf memory model is really tied to underlying arch indeed.

Some kinds of lock is the only way to guarantee the atomacity.
bpf_spin_lock is a kernel bpf helper. I guess user space could implement
the spin_lock for map element as well based on how kernel bpf_spin_lock
is implemented.

Thanks Yonghong by the info, to summarize my findings: "BPF memory model is equal to memory model of underlying architecture", cit Alexei [1], then atomicity of those kind of operations can only be guaranteed by using bpf_spin_lock in the eBPF programs and BPF_F_LOCK flag from user space applications.

[1] https://www.spinics.net/lists/netdev/msg547045.html

[2] https://www.spinics.net/lists/netdev/msg548264.html

Thanks,

Mauricio

Thanks,

Mauricio.

Thanks,

Mauricio.