Spec layer 0 — Module io_uring, Stage 3e: multi-threaded usage (shared)

Technical specification — Version 1.0 (target kernel: Linux 6.12 LTS)

Position. Stage 3e specifies the multi-threaded models for the ring. Sub-module air-sys-syscall::io_uring::shared. Builds on the invariant from ADR-022 Decision 6 (IoUring is Send but not Sync), on the IORING_SETUP_SQPOLL flag (Stage 1), ATTACH_WQ (Stage 3a) and msg_ring (Stage 2c). Reuses the core from Stage 1. No register opcode of its own.

1. The starting point: Send but not Sync

IoUring can be moved between threads (Send) but not shared by reference (!Sync). Reason: the ordering protocol from Stage 1 (§3.2) synchronises userspace ↔ kernel, not userspace ↔ userspace. The SQ/CQ heads/tails and the S1 slab are not protected against concurrent userspace accesses. Three responses, in order of Air preference.

2. Recommended model: thread-per-core (one ring per thread)

This is Air’s default model. Each thread (ideally pinned to a core) owns its own IoUring, created with SINGLE_ISSUER | DEFER_TASKRUN (Stage 1 §5.1). No sharing on the hot path, therefore no lock, low and predictable latency.

• SINGLE_ISSUER informs the kernel that a single thread submits → internal optimisations. DEFER_TASKRUN defers task-work to the wait point → fewer interruptions.
• Communication between rings: the ring is not shared; messages are passed via msg_ring (Stage 2c, MSG_DATA to wake up a peer, MSG_SEND_FD to transfer a file descriptor to it). This is the reactor-level counterpart of AirCom’s peer-to-peer model.
• Safety: trivial — no concurrent access to the same ring. The !Sync on IoUring forbids by typing accidental sharing.

This model introduces no new type: it is the direct use of IoUring, one per thread. Sections 3–5 cover cases where that is not sufficient.

3. LockedIoUring — simple sharing via a lock

#![allow(unused)]
fn main() {
pub struct LockedIoUring { /* IoUring + internal lock */ }

impl LockedIoUring {
    pub fn new(entries: NonZeroU32) -> Result<Self, Errno>;
    pub fn from_builder(builder: IoUringBuilder) -> Result<Self, Errno>;

    // Mirror API of IoUring, but taking &self (internal lock):
    pub fn submit_read(&self, fd: BorrowedFd<'_>, buffer: Vec<u8>, offset: Option<u64>)
        -> Result<SubmissionToken, Errno>;
    pub fn submit_and_wait(&self, want: u32) -> Result<u32, Errno>;
    pub fn wait_completion(&self) -> Result<Completion, Errno>;
    // … (all submit_*/completions, taking &self) …
}
// LockedIoUring: Send + Sync.
}

• A single ring shared between threads, protected by an internal lock (serialises userspace accesses to the SQ/CQ/slab). Becomes Sync.
• Warning: the lock is a contention point — under heavy multi-threaded load it serialises everything. Reserve for simple or lightly used cases. For performance, prefer thread-per-core (§2).
• Granularity: a single lock to start with (clarity, Principle 7); refinement (separate locks for submission/completion) is only considered after measurement (Principle 5), never upfront.

4. RingPool — thread-per-core helper

#![allow(unused)]
fn main() {
pub struct RingPool { /* N rings, shared io-wq, registered ring fds */ }

impl RingPool {
    /// Creates `workers` rings: the first creates the io-wq pool, the subsequent
    /// ones attach to it (ATTACH_WQ) → total number of worker threads bounded.
    pub fn new(workers: NonZeroUsize, entries: NonZeroU32) -> Result<Self, Errno>;
    /// Distributes the rings: one owned `IoUring` per worker (to be moved into
    /// each thread). Ring fds are mutually registered for msg_ring.
    pub fn into_rings(self) -> Vec<IoUring>;
    /// Routing handle to a peer ring (for msg_ring).
    pub fn handle(&self, worker: usize) -> Option<RingHandle>;
}

/// Reference to a peer ring, usable as a msg_ring target (Stage 2c).
pub struct RingHandle { /* registered ring fd of the peer */ }
}

• Role: create N coherent rings for the §2 model, by:
  • sharing the io-wq pool (ATTACH_WQ, Stage 3a) ⇒ the total number of kernel worker threads is bounded regardless of N — crucial on modest hardware (Pi 4, Charter principle 4);
  • registering ring fds mutually (Stage 3a) ⇒ msg_ring can route to a peer in a decoupled manner (wakeup, FD transfer).
• Each returned IoUring remains Send/!Sync: it is moved into its thread, never shared. RingPool creates no sharing — it organises.

5. SqpollIoUring — kernel thread polling the SQ

#![allow(unused)]
fn main() {
pub struct SqpollIoUring { /* IoUring with SETUP_SQPOLL */ }

impl SqpollIoUring {
    /// `idle`: inactivity delay before the kernel thread goes to sleep
    /// (NEED_WAKEUP). `cpu`: optional thread pinning (SQ_AFF).
    pub fn new(entries: NonZeroU32, idle: Duration, cpu: Option<u32>)
        -> Result<Self, Errno>;
    // Usual submit_* methods; submission may require NO syscall at all.
}
}

• With IORING_SETUP_SQPOLL, a kernel thread polls the SQ: in steady state, submission makes no io_uring_enter call (just the release publication). Minimal latency.
• Wakeup: if the thread has gone to sleep (after idle), the IORING_SQ_NEED_WAKEUP flag is set; the façade detects it and issues io_uring_enter(.., SQ_WAKEUP) automatically on submission (this is the only wakeup case handled on the caller’s behalf — documented).
• SQ_AFF: pins the kernel thread to a core (cpu).
• Cost: one kernel core dedicated to polling. Excellent for latency under heavy load, poor choice on Pi 4 where cores are scarce. Reserve for profiles where latency dominates and CPU is abundant (measurement, Principle 5).
• SQPOLL_NONFIXED (FEAT_SQPOLL_NONFIXED, present in 6.12) lifts the old constraint requiring exclusive use of registered FDs.

6. Send / Sync matrix

7. Safety

• The !Sync invariant on IoUring prevents by typing the sharing of an unprotected ring between threads — no data race possible on the SQ/CQ/slab.
• LockedIoUring adds the missing userspace↔userspace synchronisation; the userspace↔kernel ordering protocol from Stage 1 remains unchanged underneath.
• RingPool/thread-per-core: no sharing ⇒ no synchronisation required; all exchanges go through msg_ring (the kernel serialises writes to the target CQ).
• Teardown S2 applies per ring (each ring quiesces its own).

8. Added / shared types

New: LockedIoUring, RingPool, RingHandle, SqpollIoUring. Reuses: IoUring, IoUringBuilder, SetupFlags (SQPOLL/SQ_AFF/SINGLE_ISSUER/ DEFER_TASKRUN/ATTACH_WQ), msg_ring (Stage 2c), registered ring fd (Stage 3a).

9. Test strategy

• Thread-per-core: N threads, one ring each, independent load; hand-off via msg_ring (one thread wakes up / transfers to a peer, verified).
• LockedIoUring: concurrent access from multiple threads, completion consistency; loom on the lock + the ring protocol (no data race, no deadlock).
• RingPool: ATTACH_WQ correctly bounds the number of worker threads (observed via /proc or a counter); correct inter-ring routing.
• SqpollIoUring: syscall-free submission in steady state; correct wakeup after idle (NEED_WAKEUP); SQ_AFF pinning.
• Safety: compile-fail confirming that IoUring does not cross a & between threads (!Sync); Miri/loom for LockedIoUring.
• Coverage 100% lines + branches.

10. Key decisions that emerged at Stage 3e

1. Thread-per-core as the default model — one ring per thread, SINGLE_ISSUER + DEFER_TASKRUN, no lock; communication via msg_ring. Aligns the async runtime (ADR-023) with AirCom’s peer-to-peer model.
2. !Sync assumed on IoUring — multi-threaded safety is an explicit choice of the caller (LockedIoUring or RingPool), not a cost imposed on everyone.
3. RingPool organises, does not share — ATTACH_WQ bounds worker threads (Pi 4); rings remain owned per thread.
4. SqpollIoUring reserved — real latency gain but costs one core; discouraged on modest hardware. NEED_WAKEUP wakeup handled for the caller (the only exception to “no hidden magic”, as it is indispensable and documented).
5. Single lock first in LockedIoUring — refinement only after measurement (Principle 5).

11. Next steps

Next spec: io-uring-3f-sandbox-en.md (confinement: R_DISABLED + REGISTER_RESTRICTIONS + ENABLE_RINGS, capabilities link ADR-010/AirCom). Global English translation after validation of the French documents.

Document license: MPL 2.0 Status: Technical specification for Stage 3e (multi-threaded) of the air-sys-syscall::io_uring module, target kernel 6.12 LTS.