Explaining RX-stages for XDP

Jesper Dangaard Brouer

Let me try in a calm way (not like [1]) to explain how I imagine that
the XDP processing RX-stage should be implemented. As I've pointed out
before[2], I'm proposing splitting up the driver into RX-stages. This
is a mental-model change, I hope you can follow my "inception" attempt.

The basic concept behind this idea is, if the RX-ring contains
multiple "ready" packets, then the kernel was too slow, processing
incoming packets. Thus, switch into more efficient mode, which is a
"packet-vector" mode.

Today, our XDP micro-benchmarks looks amazing, and they are! But once
real-life intermixed traffic is used, then we loose the XDP I-cache
benefit. XDP is meant for DoS protection, and an attacker can easily
construct intermixed traffic. Why not fix this architecturally?

Most importantly concept: If XDP return XDP_PASS, do NOT pass the
packet up the network stack immediately (that would flush I-cache).
Instead store the packet for the next RX-stage. Basically splitting
the packet-vector into two packet-vectors, one for network-stack and
one for XDP. Thus, intermixed XDP vs. netstack not longer have effect
on XDP performance.

The reason for also creating an XDP packet-vector, is to move the
XDP_TX transmit code out of the XDP processing stage (and future
features). This maximize I-cache availability to the eBPF program,
and make eBPF performance more uniform across drivers.

* Instead of individual packets, see it as a RX packet-vector.
* XDP should be seen as a stage *before* the network stack gets called.

If your mind can handle it: I'm NOT proposing a RX-vector of 64-packets.
I actually want N-packet per vector (8-16). As the NIC HW RX process
runs concurrently, and by the time it takes to process N-packets, more
packets have had a chance to arrive in the RX-ring queue.

Best regards,
Jesper Dangaard Brouertho
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer

[1] https://mid.mail-archive.com/netdev@vger.kernel.org/msg127043.html

[2] http://lists.openwall.net/netdev/2016/01/15/51

[3] http://lists.openwall.net/netdev/2016/04/19/89

Join iovisor-dev@lists.iovisor.org to automatically receive all group messages.