Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-022 — Architecture of the io_uring module

Status: Accepted. Founding document of phase 0.

Category: Architecture (layer 0).

Context

io_uring is Linux’s modern asynchronous I/O mechanism, introduced in kernel 5.1 and substantially extended in subsequent versions. It offers superior performance to epoll for I/O-bound workloads and enables the expression of operations that were not possible with previous APIs (linked operations, multishot, registered buffers).

io_uring is central to Air’s architecture: it is the primary asynchronous I/O mechanism, integrated directly into layer 0 and consumed by the async runtime (ADR-023) in layer 1. The quality of the io_uring wrapper largely determines the performance and ergonomics of the Air stack.

The design of the wrapper raises several structural questions:

  • What abstraction level to adopt? Low-level (very close to the kernel) or high-level (encapsulating complexity)?
  • How to coexist with the equivalent synchronous syscalls?
  • How to manage buffers (which must remain valid for the duration of the operation)?
  • How to expose advanced features (multishot, linked, registered) without burdening the basic API?

This ADR records the 10 structural decisions made for the air-sys-syscall::io_uring module.

Decisions

Decision 1: Abstraction level 2 (typed submission/completion)

Air’s io_uring wrapper operates at abstraction level 2: submission of typed operations and retrieval of typed completions, without directly exposing the kernel ring buffers.

#![allow(unused)]
fn main() {
// Level 2 (Air default):
let token = ring.submit_read(fd, buffer, offset)?;
let completion = ring.wait_completion()?;
let bytes_read = completion.bytes_read()?;
}

Level 1 (direct manipulation of SQE/CQE) is exposed in a submodule air-sys-syscall::io_uring::raw for advanced use cases.

Rationale. Level 2 provides the ergonomics and safety required for 95% of use cases. Level 1 remains accessible for extreme optimisations and use cases that the typed API does not cover.

Decision 2: Coexistence with synchronous syscalls, shared types

Operations that have a synchronous equivalent (read, write, openat2, accept, connect, send, recv, etc.) are exposed both in the synchronous module (air-sys-syscall::fs, ::net, etc.) and in the io_uring module.

The types used are shared: SocketAddr, MessageFlags, OpenHow, etc. are the same types in both worlds. A developer who learns the synchronous API will find the same types in io_uring.

Rationale. Allows choosing the mode according to context (startup script in synchronous mode, hot path in io_uring) without relearning the API. Higher layers can progressively migrate between the two worlds.

Decision 3: Three buffer mechanisms, ownership transfer by default

For io_uring operations that read or write data, the buffer must remain valid for the entire duration of the operation (until completion). Three mechanisms are exposed:

Mechanism 1: Ownership transfer (default). The operation takes the buffer as an argument and returns it in the completion. The buffer is physically held by the ring between submission and completion.

#![allow(unused)]
fn main() {
let buffer = vec![0u8; 1024];
let token = ring.submit_read(fd, buffer, 0)?;
let completion = ring.wait_completion()?;
let (buffer, bytes_read) = completion.into_read_result()?;
}

Mechanism 2: Registered buffers (performance). Buffers are registered in advance via register_buffers, then referenced by index in operations. Avoids virtual address translation on each operation.

Mechanism 3: Raw unsafe (extreme case). The developer passes a raw pointer and guarantees the validity of the buffer. Reserved for optimisations that cannot be expressed otherwise.

Rationale. Ownership transfer is safe by construction (the compiler prevents using the buffer while the ring holds it). Advanced modes are available for cases where raw performance takes precedence over ergonomics.

Decision 4: Explicit registration, no automation

Registering FDs or buffers (register_files, register_buffers) is an explicit decision of the application. The Air wrapper makes no attempt to register automatically.

Rationale. Automation would hide costs and complicate the mental model. The developer who wants the benefit of registration requests it explicitly.

Decision 5: Multishot and linked in dedicated submodules

Multishot operations (accept multishot, poll multishot) and linked operation chains have semantics distinct from classic one-shot operations. They are exposed in dedicated submodules:

  • air-sys-syscall::io_uring::multishot for operations that produce multiple completions.
  • air-sys-syscall::io_uring::linked for operation chains.

Rationale. Clear conceptual separation. The base API remains simple; advanced APIs are available when needed.

Decision 6: IoUring is Send but not Sync, separate SharedIoUring

The primary type IoUring is Send (can be moved between threads) but not Sync (cannot be shared by reference between threads). For multi-thread use cases, a separate type LockedIoUring or SharedIoUring is exposed in the submodule air-sys-syscall::io_uring::shared.

Rationale. The majority of use cases are single-threaded (one reactor per thread, thread-per-core). Imposing Sync on the primary type would have a cost (internal lock) for the majority of cases. Multi-thread patterns are available explicitly.

Decision 7: Operations with no syscall equivalent exposed individually

Some io_uring operations have no direct syscall equivalent (for example, IORING_OP_NOP, IORING_OP_TIMEOUT, IORING_OP_LINK_TIMEOUT, IORING_OP_FILES_UPDATE). They are exposed as methods of IoUring.

Rationale. These operations are specific to io_uring and are not intended to appear in a classic syscall family. Their place is in the io_uring module.

Decision 8: Runtime detection via IORING_REGISTER_PROBE

io_uring evolves rapidly: each kernel version adds operations. The Air wrapper uses IORING_REGISTER_PROBE at startup to detect what the current kernel supports.

#![allow(unused)]
fn main() {
let ring = IoUring::new(256)?;
if ring.supports_op(IoUringOpcode::OpenAt2) {
    // use openat2 via io_uring
} else {
    // fall back to synchronous syscall
}
}

Rationale. Air must run on Linux 5.15 LTS (the reference target kernel for Debian stable, Ubuntu LTS) while allowing use of more recent features when available. Runtime detection is the proper mechanism.

Decision 9: Single Completion type, typed interpretation methods

io_uring completions contain a result (signed integer) whose semantics depend on the operation that produced the completion. Rather than exposing an enum of all possible completions (which would be enormous), Air exposes a single Completion type with typed interpretation methods:

#![allow(unused)]
fn main() {
let completion = ring.wait_completion()?;

// Depending on the type of operation being awaited:
let bytes_read = completion.bytes_read()?;        // for read
let fd = completion.accepted_fd()?;                 // for accept
let _ = completion.completed()?;                    // for close
}

The developer knows which operation they submitted and calls the appropriate method. If the operation failed, the method returns Err(Errno).

Rationale. Avoids the combinatorial explosion of a completions enum. The API remains typed and clear; the developer retains control.

Decision 10: Graceful handling of io_uring availability

If the kernel does not support io_uring (very old kernel) or if the environment has disabled it (sandbox, container), IoUring::new() returns an explicit error (Errno::ENOSYS or equivalent).

The application can then choose to fall back to synchronous syscalls, or to refuse to start.

Rationale. No silent failure. No hidden attempt at automatic fallback. The application decides.

Resulting architecture

The air-sys-syscall::io_uring module is organised into several submodules:

air-sys-syscall::io_uring::
├── (root)                      -- primary level-2 API
├── ::registration              -- register_files, register_buffers
├── ::linked                    -- linked operation chains
├── ::multishot                 -- multishot operations
├── ::shared                    -- thread-safe variants (LockedIoUring, etc.)
└── ::raw                       -- level-1 access (raw SQE/CQE)

This organisation allows progressive discovery of features: a developer starts with the root API and explores submodules when they have a specific need.

Specification phasing

The specification of the io_uring module is divided into 4 successive phases:

  • Phase 1: core API (IoUring, Completion, basic submission/completion).
  • Phase 2a: filesystem operations (21 operations: read, write, openat2, etc.).
  • Phase 2b: network operations (8 operations: accept, connect, send, recv, etc.).
  • Phase 2c: async-specific operations (9 operations: nop, timeout, etc.).
  • Phase 3a: registration (FdPool, RegisteredBuffer, ProvidedBuffers).
  • Phase 3b: linked (LinkedChainBuilder).
  • Phase 3c: multishot (accept/poll/recv multishot).
  • Phase 3d: shared (LockedIoUring, RingPool, SqpollIoUring).
  • Phase 4: raw (RawSubmissionQueueEntry, RawCompletionQueueEntry, direct ring buffer access).

This 4-phase spec is consistent with the bottom-up phasing of ADR-011.

Note (later, non-normative — does not alter any of the 10 decisions). The type names cited above for illustration (FdPool, RegisteredBuffer, ProvidedBuffers…) predate the detailed specification. The final names, aligned with ADR-029 (explicit naming), appear in the per-Stage specs and the master document: FixedFdTable, RegisteredBuffers, ProvidedBufferRing, etc. Likewise, the breakdown and operation counts (2a, 2b, 2c…) were revised and extended (adding Stages 2d, 3b, 3e, 3f) in the master document ../specs/layer-0/io-uring-0-inventaire-en.md, which is authoritative for the inventory. The soundness decisions S1/S2/S3 are recorded in ADR-028.

Future status

This ADR is immutable in its 10 decisions. Extensions to the io_uring module (new kernel operations) are added normally without amendment, as long as they respect the established conventions.

Document license: MPL 2.0 Status: Immutable founding document.