Re: The page-pool as a component for XDP forwarding

Jesper Dangaard Brouer

On Wed, 4 May 2016 09:52:08 -0700
Tom Herbert <tom@...> wrote:

On Wed, May 4, 2016 at 2:15 AM, Jesper Dangaard Brouer
<brouer@...> wrote:

I've started a separate document for designing my page-pool idea.

I see the page-pool as a component, for allowing fast forwarding with
XDP, at the packet-page level, cross device.

I want your input on how you imagine XDP/eBPF forwarding would work?
I could imagine,
1) eBPF returns an ifindex it want to forward to,
2) look if netdevice supports new NDO for XDP-page-fwd
3A) call XDP-page-fwd with packet-page,
3B) No XDP-page-fwd, then construct SKB and xmit directly on device,
4) (for both above cases) later at TX-DMA completion, return to page-pool.

Below I propose that we use XDP eBPF in a new fashion, based on the
state of the feedback loop that the page-pool offer. With eBPF hooks
at both RX and page-return-time, we can implement a super powerful DDoS
protection mechanism. Does it make sense?
Mostly ;-). I like the idea of returning an index from eBPF which
basically just gives a queue to transmit. Presumably, each receive
queue would have it's own XDP transmit queue that it can use lockless.
Also, I think it is reasonable that we could cross devices but within
the _same_ driver (like supporting forwarding between two Mellanox
NICs). In that case each RX queue has one dedicated XDP TX queue for
each device.
I'm not sure how you can get lockless TX with only one XDP-TX queue.

Remember we have to build in bulk TX from day one. Why? Remember the
TX tailptr write is costing in the area of 100ns. Thus, too expensive
to send single TX frames.

For forwarding on a non XDP queue (like 3B or crossing different type
devices) I don't think we should do anything special. Just pass the
packet to the stack like in the olden days and use the stack
forwarding path. This is obviously slow path, but not really very
interesting to optimize in XDP.
Yes for 3B.
For 3A I want to support cross device driver.

One thing I am not sure how to deal with is flow control. i.e. if the
transmit queue is being blocked who should do the drop. Preferably,
we'd want the to know the queue occupancy in BPF to do an intelligent
drop (some crude fq-codel or the like?)
Flow control or push-back is an interesting problem to solve.

Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of

Designing the page-pool
:Version: 0.1.1
:Authors: Jesper Dangaard Brouer


Motivation for page recycling is primarily performance related (due to
network stack use-cases), as bottlenecks exist in both the page
allocator and DMA APIs.

It was agreed at MM-summit 2016, that the ideas developed to speedup
the per CPU caching layer, should be integrated into the page
allocator, where possible. And we should try to share data
structures. (

The page-pool per device, still have merits, as it can:
1) solve the DMA API (IOMMU) overhead issue, which
2) in-directly make pages writable by network-stack,
3) provide a feedback-loop at the device level

Referring to MM-summit presentation, for the DMA use-case, and why
larger order pages are considered problematic.

MM-summit 2016 presentation avail here:

XDP (eXpress Data Path)

The page-pool is a component that XDP need in-order to perform packet
forwarding at the page level.


Avoid NUMA problems, return to same CPU

A classical problem, especially for NUMA systems, is that in a
asymmetric workload memory can be allocated on one CPU but free'ed on
a remote CPU (worst-case a remote NUMA node). (Thus, CPU "local"
recycling on free is problematic).

Upfront, our design solves this issue, by requiring pages are recycled
back to the originating CPU. (This feature is also beneficial for the
feedback-loop and associated accounting.)

Reduce page (buddy) fragmentation

Another benefit of a page-pool layer on-top of a driver, which can
maintain a steady-state working-set, is that the page allocator have
less chances of getting fragmented.

Feedback loop

With drivers current approach (of calling the page allocator directly)
the number of pages a driver can hand-out is unbounded.

The page-pool provide the ability to get a feedback loop facility, at
the device level.

A classical problem is that a single device can take up an unfair
large portion of the shared memory resources, if e.g. an application
(or guest VM) does not free the resources (fast-enough). Thus,
negatively impacting the entire system, possibly leading to
Out-Of-Memory (OOM) conditions.

The protection mechanism the page-pool can provide (at the device
level) MUST not be seen as a congestion-control mechanism. It should
be seen as a "circuit-breaker" last resort facility to protect other
parts of the system.

Congestion-control aware traffic usually handle the situation (and
adjust their rate to stabilize the network). Thus, a circuit-breaker
must allow sufficient time for congestion-control aware traffic to

The situations that are relevant for the circuit-breaker, are
excessive and persistent non-congestion-controlled traffic, that
affect other parts of the system.

Drop policy

When the circuit-breaker is in effect (e.g. dropping all packets and
recycling the page directly), then XDP/eBPF hook could decide to
change the drop verdict.

With the XDP hook in-place, it is possible to implement arbitrarily
drop policies. If the XDP hook, gets the RX HW-hash, then it can
implement flow based policies without touching packet data.

Detecting driver overload

It might be difficult to determine when the circuit-breaker should
kick-in, based on an excessive working-set size of pages.

But at the driver level, it is easy to detect when the system is
overloaded, to such an extend that it cannot process packets
fast-enough. This is simply indicated by the driver cannot empty the
RX queue fast-enough, thus HW is RX dropping packets (FIFO taildrop).

This indication could be passed to a XDP hook, which can implement a
drop policy. Filtering packets at this level can likely restore
normal system operation. Building on the principal of spending as few
CPU cycles as possible on packets that need to be dropped anyhow (by a
deeper layer).

It is important to realize that, dropping the the XDP driver level is
extremely efficient. Experiments show that, the filter capacity of
XDP filter is 14.8Mpps (DDIO touching packet and updating up an eBPF
map), while iptables-raw is 6Mpps, and hitting socket limit is around
0.7Mpps. Thus, an attacker can actually consume significant CPU
resources by simply sending UDP packets to a closed port.

Performance vs feedback-loop accounting

For performance reasons, the accounting should be kept as per CPU

For NIC drivers it actually makes sense to keep accounting 100% per
CPU. In essence, we would like the circuit-breaker to kick-in per RX
HW queue, as that would allow remaining RX queue traffic flow.

RX queues are usually bound to a specific CPU, to avoid packet
reordering (and NIC RSS hashing (try-to) keep flows per RX queue).
Thus, keeping page recycling and stats per CPU structures, basically
achieves the same as binding a page-pool per RX queue.

If RX queue SMP affinity change runtime, then it does not matter. A
RX ring-queue can contain pages "belonging" to another CPU, but it
does not matter, as eventually they will be returned to the owning

It would be possible to also keep a more central state for a
page-pool, because the number of pages it manage only change when
(re)filling or returning pages to the page allocator, which should be
a more infrequent event. I would prefer not to.

Determining steady-state working-set

For optimal performance and to minimize memory usage, the page-pool
should only maintain the number of pages required for the steady-state

The size of the steady-state working-set will vary depending on the
workload. E.g. in a forwarding workload it will be fairly small.
E.g. for a TCP (local)host delivery workload it will be bigger. Thus,
the steady-state working-set should be dynamically determined.

Detecting steady-state by realizing, that in steady state, no
(re)filling have occurred for a while, and the number of "free" pages
in the pool is not excessive.

Idea: could we track number of page-pool recycle alloc and free's
within N x jiffies, and if the numbers (rate) are approx the same,
record number of outstanding pages as the steady-state number? (Could
be implemented as single signed counter reset every N jiffies, inc/dec
for alloc/free, approaching zero (at reset point) == stable)

If RX rate is bigger than TX/consumption rate, queue theory says a
queue will form. While the queue builds (somewhere outside out our
control), the page-pool need to request more and more pages from
page-allocator. The number of outstanding pages increase, seen from
the page-pool, proportional to the queue in the system.

This, RX > TX is an overload situation. Congestion-control aware
traffic will self stabilize. Assuming dealing with
non-congestion-controlled traffic, some different scenarios exist:

1. (good-queue) Overload only exist for a short period of time, like a
traffic burst. This is "good-queue", where we absorb bursts.

2. (bad-queue) Situation persists, but some other limit is hit, and
packets get dropped. Like qdisc limit on forwarding, or local
socket limit. This could be interpreted as a "steady-steady", as
page-recycling reach a certain level, and maybe it should?

3. (OOM) Situation persists, and no natural resource limit is hit.
Eventually system runs dry of memory pages and OOM. This situation
should be caught by our circuit-breaker mechanism, before OOM.

4. For forwarding, the hole code path from RX to TX, takes longer than
the packet arrival rate. Drops happen at HW level by overflowing
RX queue (as it is not emptied fast enough). Possible to detect
inside driver, and we could start a eBPF program to filter?

After an overload situation, when RX decrease (or stop), so RX < TX
(likely for a short period of time). Then, we have the opportunity to
return/free objects/pages back to the page-allocator.

Q: How quickly should we do so (return pages)?
Q: How much slack to handle bursts?
Q: Is "steady-state" number of pages an absolute limit?

XDP pool return hook

What about allowing a eBPF hook at page-pool "return" point? That
would allow eBPF to function as an "egress" meter (in circuit-breaker

The XDP eBPF hook can maintain it's own internal data structure, to
track pages.

We could saved the RX HW hash (maybe in struct-page), then eBPF could
implement flow metering without touching packet data.

The eBPF prog can even do it's own timestamping on RX and compare at
pool "return" point. Essentially implementing a CoDel like scheme,
measuring "time-spend-in-network-stack". (For this to make sense, it
would likely need to group by RX-HW-hash, as multiple ways through the
netstack exist, thus it cannot be viewed as a FIFO).


The resource limitation/protection feature offered by the page-pool,
is primarily a circuit-breaker facility for protecting other parts of
the system. Combined with a XDP/eBPF hook, it offers a powerful and
more fine-grained control.

It requires more work and research if we want to react
"earlier". e.g. before the circuit-breaker kicks in. Here one should
be careful not to interfere with congestion aware traffic, by giving
it sufficient time to reach.

At the driver level it is also possible to detect, if system is not
processing RX packets fast-enough. This is not an inherent feature of
the page-pool, but it would be useful input for a eBPF filter.

For the XDP/eBPF hook, this means that it should take a "signal" as
input of how the current operating machine state is.

Considering the states:
* State:"circuit-breaker"- eBPF can choose to approve packets, else stack drop
* State:"RX-overload" - eBPF can choose to drop packets to restore operation

Relating to page allocator

The current page allocator have a per CPU caching layer for
order-0 pages, called PCP (per CPU pages) ::

struct per_cpu_pages {
int count; /* number of pages in the list */
int high; /* high watermark, emptying needed */
int batch; /* chunk size for buddy add/remove */

/* Lists of pages, one per migrate type stored on the pcp-lists */
struct list_head lists[MIGRATE_PCPTYPES];

The "high" watermark, can be compared to (dynamic) steady-state
number, which determine how many cached (order-0) pages are kept,
before they are returned to the page allocator.

For PCP once the "high" watermark is hit, then "batch" number of
pages are returned. (Using a batch (re)moves the pathological
case of two object working-set being recycles on the "edge" of
the "high" watermark, causing too much interaction with the page

On my 8 core (i7-4790K CPU @ 4.00GHz) with 16GB RAM, the values
for PCP are high=186 and batch=31 (note 31*6 = 186). These
setting are likely not optimal for networking, as e.g. TX DMA
completion is default allowed to freeing up-to 256 pages.

The question is, whether the PCP "high" watermark could be
dynamically determined by the same method proposed for
determining the steady-state criteria?

Background material

Circuit Breaker

Quotes from:

.. _RFC-Circuit-Breaker:

RFC-Circuit-Breaker_ ::

[...] non-congestion-controlled traffic, including many applications
using the User Datagram Protocol (UDP), can form a significant
proportion of the total traffic traversing a link. The current
Internet therefore requires that non-congestion-controlled traffic is
considered to avoid persistent excessive congestion

RFC-Circuit-Breaker_ ::

This longer period is needed to provide sufficient time for transport
congestion control (or applications) to adjust their rate following
congestion, and for the network load to stabilize after any

RFC-Circuit-Breaker_ ::

In contrast, Circuit Breakers are recommended for non-congestion-
controlled Internet flows and for traffic aggregates, e.g., traffic
sent using a network tunnel. They operate on timescales much longer
than the packet RTT, and trigger under situations of abnormal
(excessive) congestion.

Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of

Join to automatically receive all group messages.