On Wed, Mar 27, 2019 at 1:48 PM Pablo Alvarez via Lists.Iovisor.Org
<palvarez=akamai.com@...> wrote:

Why is it required that llvm compile the BPF code with -O2? That seems
to be part of what is causing these verifier problems...

-O0 won't work as helper call will become an indirect call
static void *(*bpf_map_lookup_elem)(void *map, void *key) =
(void *) BPF_FUNC_map_lookup_elem;
Another reason is below value tracking in verifier.

The verifier does not handle value spill well. It kept track of map
pointers, packet pointers, etc.
But if a particular value is spilled, later on, when reload from
stack, it will become unknown.
For example,
r1 = 10; /* verifier state: r1 = 10 */
*(r10 - 40) = r1;
r2 = *(r10 - 40); /* verifier state: r2, unknown */

The reason is that keep tracking of values could increase verification
time quite a bit.
This is measured sometime back, but it may warrant to do some measurement again
at this moment to see whether can relax this restriction.

So practically -O0 won't work. -O1 may or may not work and most people
do not use it.
-O2 is preferred. Also for performance reasons, we want to make -O2
work as BPF JIT
inside the kernel does not do any optimization, it is simple one insn
each time translation.

On 3/27/19 4:23 PM, Yonghong Song wrote:

On Wed, Mar 27, 2019 at 10:17 AM Jiong Wang <jiong.wang@...> wrote:

On 27 Mar 2019, at 16:43, Simon <contact@...> wrote:

Thx a lot for your time Jiong.

The more I played with bpf/xdp, the more I understand that the challenge is about making "optimized byte code" compliant for the verifier.

How could I do this kind of checks my self ? I mean looking how llvm optimized my code ? (to be able to do same kind of analyses you do above?)

Just my humble opinion, I would recommend:

1. get used to verifier rejection information, for example:

R0=inv1 R1=pkt(id=0,off=0,r=42,imm=0) R2=pkt_end(id=0,off=0,imm=0) R3=inv(id=0) R4=inv(id=0,umax_value=504,var_off=(0x0; 0x1ff)) R5=inv5 R10=fp0,call_-1
40: (0f) r1 += r3
math between pkt pointer and register with unbounded min value is not allowed

It tells you the status of each registers at the rejection point,
for example, now R3 is “inv”, meaning a scalar value (not a pointer),
and is without value range, then r4 has value range, and maximum value
is 504.

If you use BPF constructor debug=16 flag, it will print out the
register state for every insn if you are even more curious.

2. known what verifier will reject. Could refer to:

https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/tree/tools/testing/selftests/bpf/verifier?id=473c5daa86ffe91e937856cc32b4faa61db2e3e3

those are unit examples of what will be rejected, and some of them are with
meaningful test name or comments so could be easy to understand.

To resolve this issue, llvm may need to do more:
- prevent/undo optimization which may cause ultimate verifier rejections.
- provide hints (e.g., through BTF) to verifier so verifier may
selectively do some analysis
or enable some tracking for the cases where BTF instructed to
handle. For example,
BTF may tell verifier two register have the same state at a
particular point and verifier
only needs to check these two registers with limited range and no
others, etc.

Regards,
Jiong