rfc: 0008 title: Write-ahead log — durable buffer between OTLP receiver and Parquet writer status: accepted author: Jens Holdgaard Pedersen jens@holdgaard.org drafting-assistance: Claude created: 2026-05-29 supersedes: — superseded-by: —

RFC 0008 — Write-ahead log

Status note. accepted (2026-06-14, maintainer sign-off — the terminal ladder status per docs/rfcs/README.md). Reached green 2026-06-13 (criteria below); validated is vacuous for the WAL (its pillar touches no thesis gate — see the closing paragraph), so the maintainer advances it directly from green to accepted. The docs/verification.md §3 / docs/rfcs/README.md ladder defines green as all §5 acceptance criteria pass. Every RFC0008 §5 arm (.1–.10) now has a live, passing test — no #[ignore]/unimplemented!() acceptance stubs remain under crates/ourios-wal/tests/ (the encode_audit_event free fn in lib.rs is still unimplemented!(), but that is the §9 AuditEvent serde-format deferral, not a §5 acceptance criterion): .1 wal-before-ack, .2 crash-recovery completeness (the real-SIGKILL CI gate), .3 recovery non-amplification, .4 torn-write heal, .5 corruption (all five reasons), .6 segment rotation, .7 checkpoint + durable sidecar, .8 batched-fsync group commit, .9 bounded wal_unflushed_bytes, .10 the startup recovery driver. Landed across #123/#126 (recovery), #185–#187 (snapshot restore), #188 (rotation + §6.9 cadence), #189 (this spec’s audit-deferral amendment), #190 (the .1/.3/.4/.5/.9 arms), and #191 (.8).

One deferral, not a gap. RFC0008.5’s corruption audit event is deferred to a system-scoped-audit follow-up (§9, 2026-06-13 amendment) — the audit stream is tenant-partitioned and WAL corruption has no tenant. The arm’s load-bearing halves (structured RecoveryError + halt-all-segments) are green and corruption stays loud (wal_corrupt_frames_total); only the durable queryable record is postponed.

validated is vacuous here; accepted is the maintainer’s call. The ladder’s validated stage gates on thesis-gates in benchmarks.md §7 (compression / query-latency / reconstruction). The WAL is a durability buffer — its pillar touches no thesis gate — so the validated condition (“every thesis-gate the RFC’s pillars touch passes”) is vacuously satisfied now that §5 is green. The WAL never self-promoted through the vacuous validated stage; the terminal accepted flip was the maintainer’s call per docs/rfcs/README.md, granted 2026-06-14.

How to read this document. Sections §§1–4 are the design contract — the what and the why. §5 lists the normative Given / When / Then scenarios — the contract. §6 is the precise specification the ourios-wal crate is implemented against. §7 records the alternatives we evaluated and rejected. §8 maps each §5 scenario to a test technique and a test file. §9 holds the open questions still up for debate.

1. Summary

Ourios introduces an ourios-wal crate that owns the on-disk write-ahead log between the OTLP receiver and the rest of the ingest pipeline. Every accepted OTLP batch lands as one length-prefixed, CRC-validated frame in an append-only segment file on local disk; the receiver acknowledges only after the batch is durably fsync’d, satisfying CLAUDE.md §3.4 WAL-before-ack. Segments rotate by size or time (whichever first); a recovery scanner replays surviving frames through the normal ingest pipeline on restart; and a checkpoint mechanism lets the Parquet writer signal which records are durably on object storage so the corresponding WAL segments can be deleted. Replication is explicitly out of scope — when it lands, it is in addition to the WAL, not instead of it (CLAUDE.md §3.4).

2. Motivation

2.1 The §3.4 invariant has no implementation

CLAUDE.md §3.4 pins WAL-before-ack as one of the non-negotiable invariants (“Ingester acknowledges an OTLP batch only after it has been durably written to the WAL. … No in-memory-only acks, ever.”). docs/hazards.md H3 carries the matching hazard. No crate currently implements it. RFC 0003 §6.5 calls the contract out as a hard dependency (“the receiver itself is post-MVP per roadmap.md §5; … cannot be enabled until ourios-wal lands, and there is no MVP code path that acks a network request before durability”), but leaves the WAL itself unspecified. This RFC is that specification.

2.2 The receiver is blocked on this

RFC 0003 (OTLP gRPC + HTTP receiver) is at drafted and will stay there until §5’s acceptance criteria can credibly assert the WAL-before-ack sequence — which requires the WAL to exist. Landing ourios-wal therefore unblocks RFC 0003’s progression to specified, then implementation, then a real “telemetrygen → ourios-receiver → measure” live-services test path. The bench’s PR-K2 file-loader path (RFC 0006 §3.1) remains the MVP route in the meantime; the WAL is what lets ingest happen over the wire.

2.3 Roadmap context

docs/roadmap.md §5 lists the WAL as post-MVP. This RFC does not move it into the MVP; it specifies the post-MVP implementation. The MVP corpus path (file loader → miner → Parquet writer) does not use the WAL — it bypasses the receiver entirely, which is precisely why the bench works today without the WAL.

3. Background — what we are and aren’t building

3.1 What a WAL is in this context

A write-ahead log is the canonical pattern from durable storage systems (PostgreSQL, RocksDB, LMDB, Kafka): every durability-relevant change is appended to a strictly-sequential log on stable storage before it is acknowledged or applied to the live data structures. On crash, the system replays the log to reach a consistent post-crash state. Two properties make WALs suitable for the §3.4 invariant:

1. Sequential append + batched fsync amortises the disk sync cost across many small writes — orders of magnitude cheaper than per-record fsync, with the durability still bounded by the batch latency.
2. Strict ordering lets recovery be a forward scan: read frames left-to-right and replay each. No log-side rewrite, no merge.

We are not building Raft / Paxos / Chubby. The WAL is a single-writer single-node component; replication is CLAUDE.md §3.4’s explicit non-goal at this layer.

3.2 What goes into the WAL

The WAL carries two frame kinds, distinguished by a 1-byte discriminator (§6.2.2):

• FrameKind::OtlpBatch — payload is the OTLP protobuf bytes the wire delivered (ExportLogsServiceRequest), verbatim. Per-batch (not per-record) granularity matches the §3.4 ack boundary; one fsync gates one ack. The payload is the same bytes the receiver decoded, so recovery is “re-run the decode + fan-out pipeline” (§6.6) rather than “rehydrate per-record proto from a custom format.” Tenant fan-out (RFC 0003 §6.3) stays on the receiver’s side of the WAL.
• FrameKind::AuditEvent — payload is a single ourios_core::audit::AuditEvent (the TemplateWidened / TemplateTypeExpanded / TemplateWideningRejectedDegenerate shape RFC 0001 §6.4 defines). RFC 0001 §6.7 and RFC 0005 §3.7 both pin “audit events route through the same WAL path as data records;” RFC 0005 §3.7 explicitly defers the contract to this RFC (“a contract that lands with the post-MVP ourios-wal crate; until then audit-event durability is in-memory and the corpus bench accepts that”). The frame- kind discriminator is how that contract is honoured. The exact AuditEvent byte encoding is §9 open (“AuditEvent has no serde derives today; the encoder lands alongside them”).

The receiver flow RFC 0003 §6.5 already pins for the OTLP-batch kind: decode batch → WAL append → fsync → ack on the critical path; tenant fan-out + mining + Parquet writes happen post-ack. The audit-event kind is appended during the post-ack mining work — under the §6.4 audit-ordering barrier: audit-event frames for record R must be append-ed and sync-ed before the data Parquet row group containing R becomes durable on object storage. RFC 0001 §6.4 spells this out as a normative requirement on this RFC (“a data record carrying template_version = V must not become durable before every audit event justifying the leaf’s progression to V is durable”); §6.4 below pins the WAL-side discipline that satisfies it.

3.3 What does not go into the WAL

• Miner state. The Drain tree is reconstructed by replaying the WAL records through the miner; we do not snapshot it. Snapshotting becomes interesting if recovery time on a hot-template corpus exceeds the operator’s patience; that is §9 open question.
• Parquet writer state. The Parquet writer’s open row groups are not durable; on crash the in-flight row group is lost and rebuilt from the WAL. Atomic-publish (RFC 0005 §3.7 / RFC0005.2) means closed row groups are durable and need no replay.
• Query state. The querier is read-only against object storage; it has no recovery concern at this layer.

4. Background — existing Rust durability ecosystem

4.1 What we surveyed

Rust’s WAL crate landscape divides into two camps:

• Heavy embedded databases (sled, redb, fjall, rocksdb via FFI). These give us a WAL for free but also a B-tree, MVCC, transaction model, and a persistent key-value abstraction — none of which the ingest path consumes. Carrying them adds dependency surface area (sled is still pre-1.0 with known durability bugs; rocksdb adds a C++ link and forecloses pure-Rust toolchains) for an API we do not use.
• Focused WAL crates (okaywal, wal, vlog). Smaller surface, but each makes a different opinionated choice (multi-writer arbitration, async vs sync I/O, segment format) and the format becomes our wire-compat boundary the same way it would for a hand-rolled implementation.

4.2 Why hand-roll

The component we need is small (single writer, append-only, length-prefixed framing, CRC, batched fsync, segment rotation, linear recovery, checkpoint-driven truncation — call it 600 lines of careful Rust). The wire format is what we have to be careful about; the surrounding code is mechanical. A focused crate would impose its own format conventions on us at the same boundary, so the dependency saves implementation time but not design time. We hand-roll, keep the format under our direct control, and §7 records the focused-crate alternatives we considered.

5. Acceptance criteria

Scenario RFC0008.1 — WAL-before-ack [§3.4]

• Given a fake receiver wired to a real Wal (opened with defaults) and a single OTLP ExportLogsServiceRequest batch ready to acknowledge
• When the receiver runs its accept path
• Then the 2xx (gRPC OK) response is emitted only after the Wal::sync call returns Ok(_) — measured by an AtomicBool set after sync returns Ok(_) (not inside sync; the probe inside sync would already be true mid-call, which would let an ack racing the sync trivially pass). The ack-emit path asserts the flag is true before sending the response, and the receiver’s pre-sync points (decode, tenant_derive, append) all assert the flag is false
• And wal_unflushed_bytes is non-zero between append and sync, and zero after sync returns
• And a fault injected at append (returns AppendError) suppresses both the sync and the ack (no record of the batch ever reaching disk or the client)
• And a fault injected at sync (returns SyncError) suppresses the ack and surfaces the error to the receiver per §9’s “what does the receiver do when sync returns error?” open question — the test asserts that the ack does not fire, the specific receiver behaviour beyond that being out of this RFC

Scenario RFC0008.2 — Crash-recovery completeness [§3.4 / H3]

• Given a cargo test harness that forks a child running a receiver wired to a real Wal, with the parent controlling the request rate and a deterministic record generator
• When the parent sends SIGKILL between Wal::append and Wal::sync (first arm), and between Wal::sync and the simulated ack (second arm)
• Then on restart, Wal::replay produces every frame the child had fsync’d before the kill (no fsync’d frame is lost — the H3 invariant)
• And any frame whose append completed but sync did not falls into one of three buckets, per §6.6: replayed as a normal frame (complete + CRC-valid → the kernel post-mortem flush left it on disk; legitimate but un-acked at the client), surfaced via the §6.6 step 4 newest-segment torn-tail truncate (partial header / partial payload), or surfaced as RFC0008.5 corruption (complete frame whose CRC32-C doesn’t match — even on the newest segment, a CRC mismatch is corruption, not truncation). The test does not assert “exactly the fsync’d frames” — that would be stronger than SIGKILL semantics admit, since process death does not discard dirty page-cache data
• And FrameKind::AuditEvent frames the miner had appended-and-fsync’d before the kill survive identically (per the RFC 0005 §3.7 / RFC0005.2 audit-durability contract this RFC implements via §6.4)
• And this scenario’s test runs on every PR; failure blocks merge

Scenario RFC0008.3 — Crash-recovery non-amplification [H3]

• Given a fixture corpus of N ∈ {1, 4, 16} segments each at the default wal_segment_size_bytes (128 MiB) of representative OtlpBatch payloads, captured by a criterion benchmark
• When Wal::replay runs over the directory and the benchmark measures wall time
• Then total wall time scales O(N) in segment count, within a ±20 % tolerance to allow for warm-cache vs cold-cache differences across the runs
• And wal_syncs_total does not advance during replay (replay is read-only on closed segments; the §6.6 step 4 heal fsync only fires on a torn-tail newest segment, which the benchmark fixture deliberately omits)
• And no per-record audit event is emitted by the recovery driver itself (the happy path emits none; the single RFC0008.5 corruption-path audit event is deferred per the 2026-06-13 §5 amendment, so today the happy path and the corrupt path alike emit zero)

Scenario RFC0008.4 — Torn last frame on the newest segment [H3]

• Given a segment directory where the lexicographically- greatest segment file has a partial frame at its tail — either a partial 12 B frame header, or a valid header followed by a short payload (truncated at a random offset inside the payload bytes)
• When Wal::replay walks that directory in order
• Then the partial bytes are treated as clean truncation: the scan stops on this segment, no error is returned, no audit event is emitted
• And §6.6 step 4 heals the segment: ftruncate(2) to the last valid frame boundary recorded during the scan, fdatasync the segment, fsync the parent directory — the truncated bytes are physically gone from disk
• And the post-recovery Wal::append lands at the frame boundary preceding the torn write
• And a partial header / payload on any older (non- newest) segment is not treated as truncation — it surfaces as RFC0008.5 corruption, because older segments are post-rotation and their final fsync completed; a torn tail there is file-system corruption rather than a legit rotate-pending-fsync. This newest-vs-older fork is the scenario’s central pin

Scenario RFC0008.5 — Corrupt frame [H3]

• Given a segment with one of: a CRC32-C mismatch on a complete frame, an unknown kind byte (> 0x02), a non-zero _pad, an oversize len (> MAX_FRAME_BYTES), or a torn header/payload on a closed (non-newest) segment
• When Wal::replay walks the segment containing the corruption
• Then the scan emits a structured RecoveryError carrying the segment UUIDv7 + byte offset of the corrupt frame
• And recovery stops scanning all segments — the high-water-mark logic depends on contiguous ordering and a corrupt frame invalidates everything past it (an operator must intervene before the receiver opens its listeners)
• And the test exercises all five corruption sub-cases in separate arms (crates/ourios-wal/tests/corruption.rs)

Amendment 2026-06-13 (corruption audit event deferred). The original criterion also required a WalRecoveryCorruption { segment, offset, reason } audit event appended to the audit-event Parquet writer’s queue. That half is deferred to a follow-up (§9): the RFC 0005 audit stream is tenant-partitioned (CLAUDE.md §3.7) and WAL corruption is a system event with no tenant, so the durable forensic record needs a system-scoped-audit design that does not yet exist. Deferring it costs no safety — corruption is already loud: replay returns the structured RecoveryError and recovery halts, so the receiver refuses to open its listeners and an operator must intervene. Only the queryable record is postponed, not the failure response. RFC0008.5 is satisfied by the structured-error + halt-all-segments halves; wal_corrupt_frames_total (§6.8) already counts occurrences for the operator dashboard.

Scenario RFC0008.6 — Segment rotation

• Given an open Wal at default wal_segment_size_bytes (128 MiB) and wal_segment_age_secs (600 s)
• When an append would push the current segment past the size cap (size-cap arm), OR wal_segment_age_secs elapses since the current segment’s header was written (time-cap arm), whichever fires first per §6.5
• Then the current segment is closed cleanly: the final fdatasync on the segment + fsync on the parent directory complete before any frame is appended to the new segment
• And a fresh segment is opened with a new UUIDv7 filename, the 24 B segment header is written, and the subsequent append lands in the new segment — not the old one
• And no frame is dropped or duplicated across the rotation boundary (asserted by counting frames before / after rotation in the test)
• And a rotation whose final fsync returns an error surfaces as a hard AppendError: subsequent append calls return the same error, the receiver refuses to accept new batches, and the operator must intervene

Scenario RFC0008.7 — Checkpoint-driven truncation + durable sidecar

• Given a Wal that has accumulated K segments and a simulated Parquet writer that has durably committed records up to WalOffset X
• When Wal::checkpoint(X) runs followed by a housekeeping pass
• Then the <wal_root>/CHECKPOINT sidecar is atomically written (CHECKPOINT.tmp → fsync → rename → parent- dir fsync), containing the 32 B record {magic = "OWCK", version = 1, flags = 0, segment_uuid (16 B), byte (8 B LE)}
• And every segment whose highest frame offset is ≤ X is unlink’d; segments straddling X are kept
• And wal_disk_bytes drops by the sum of the deleted segments’ sizes

Crash-between-checkpoint-and-housekeeping arm:

• Given the same setup
• When SIGKILL fires between Wal::checkpoint(X) returning and the housekeeping pass starting
• Then the CHECKPOINT sidecar survives the crash (the atomic-write + fsync above is what guarantees this)
• And on restart, Wal::last_checkpoint() returns Some(X), and the recovery driver suppresses every frame at append-offset ≤ X on its Parquet path — records already published are not re-fed and not duplicated on the data side (replay is at-least-once; dedup is explicitly out of scope, so this suppression is the only mechanism that prevents the dup). replay itself still delivers those frames — the miner may need them when its snapshot lags (2026-06-12 amendment; suppression is per consumer, see §6.6)

Surviving-segments / no-global-counter arm:

• Given a checkpoint has advanced past several older segments and housekeeping has unlinked them
• When a fresh Wal::open is followed by a new Wal::checkpoint(Y) where Y > X
• Then the operation proceeds against the surviving UUID-named files without reconstructing a global offset counter — the (segment_uuid: UUIDv7, byte) WalOffset representation makes this trivially well-defined

Retain-floor arm (2026-06-12 amendment):

• Given a checkpoint at X and a retain floor S < X (a miner snapshot lagging the Parquet writer)
• When Wal::housekeeping(Some(S)) runs
• Then only segments whose highest frame offset is ≤ both X and S are unlinked — segments holding frames in (S, X] survive, because the miner snapshot has not captured them yet and truncating them would leave a hole in the restored tree (template drift, hazard #5)
• And a later pass with an advanced floor S' ≥ X reclaims them — the floor delays truncation, never cancels it

Scenario RFC0008.8 — Batched-fsync knob

• Given the group-commit CommitCoordinator over a Wal, with wal_batch_window_ms set in turn to {10, 100, 1000}
• When a batch of commits is fired together under a paused virtual clock — so the only time that elapses is the coordinator’s own window timer; the real fsync runs but does not advance the virtual clock
• Then each commit’s ack latency equals the configured window exactly (deterministic, no scheduler/instrumentation jitter) — the window is the dominant contributor, not per-record fsync cost (which would ack at ≈ 0 independent of the window). This virtual-clock formulation supersedes the original wall-clock-P99-within-±30 % one, which was non-deterministic on shared and instrumented CI runners.
• And wal_syncs_total advances at a rate ≈ (steady-state arrival-rate / per-window batch size) rather than per-record (so appends_per_sync is well above 1 in every setting)
• And the §3.4 invariant holds across all three: no ack precedes the Wal::sync whose returned offset covers the corresponding frame

Scenario RFC0008.9 — wal_unflushed_bytes is bounded [H3 detection]

• Given a Wal under a randomised arrival pattern (proptest-driven mix of small frames + max-sized 16 MiB frames + small frames, varying interleavings)
• When the test samples wal_unflushed_bytes at every append and sync boundary across the run
• Then the metric never exceeds 2 × wal_segment_size_bytes at any sample, regardless of arrival rate
• And the §6.9 tunables-table lower bound on wal_segment_size_bytes (≥ MAX_FRAME_BYTES + segment_header + frame_header, validated as ≥ 17 MiB) is what makes the bound achievable: a max-sized frame always fits inside a single segment, so the in-flight batch can never straddle three segments at once
• And a configuration with wal_segment_size_bytes below the lower bound is rejected at Wal::open time (OpenError::InvalidConfig) — the validation runs on every §6.9 tunable, not just this one

Scenario RFC0008.10 — Startup recovery driver: per-consumer horizons (2026-06-12 amendment)

• Given a WAL whose frames span a CHECKPOINT at X and a durable miner snapshot whose recorded high-water mark is S ≥ X (the steady-state ordering: snapshots are taken at segment rotation, the Parquet checkpoint lags)
• When the ingester starts
• Then recovery runs to completion before the network listeners open (no live append interleaves with replay)
• And suppression is per consumer, by each delivered frame’s offset: the Parquet path consumes only frames > X (frames ≤ X are already on object storage), the miner only frames > S (the restored snapshot already covers (X, S], so re-feeding them would double-apply)
• And the post-recovery miner tree state is equal to a control tree built by ingesting the same records from scratch (the RFC 0001 §3.5.3 equivalence, observed through this driver)

Lagging-snapshot catch-up arm (S < X):

• Given the same WAL but a snapshot whose mark S is below the checkpoint X, with the (S, X] frames retained on disk by the §6.7 floor
• When the ingester starts
• Then the miner consumes the retained (S, X] frames (closing its state gap — the floor’s payoff) while the Parquet path suppresses them (already published), and both consume frames > X
• And the post-recovery tree again equals the from-scratch control

Snapshot-cadence arm:

• Given live ingest crossing a segment-rotation boundary
• When the rotation completes
• Then a per-tenant snapshot write is triggered, recording the rotation-point high-water mark (RFC 0001 §6.9 cadence), and the housekeeping retain floor advances to the new snapshot’s mark once the write is durable

6. Proposed design

6.1 Overall shape

Amendment 2026-06-12 (snapshot-restore support). Four changes land together with RFC 0001 §6.9’s v2 restore (the same-day amendment there is the other half of this design): FrameSink::consume now receives the frame’s WalOffset, and replay delivers every well-formed surviving frame — suppression moves out of replay and into the recovery driver as per-consumer replay horizons (the Parquet path consumes frames above the checkpoint, the miner frames above its snapshot’s high-water mark — §6.6; an in-replay skip would make a lagging snapshot’s retained frames undeliverable); the checkpoint offset is exposed to the driver as last_checkpoint; the housekeeping pass becomes an explicit API taking an optional retain floor so truncation never outruns the miner snapshot (§6.7); and replay takes &mut self, matching the landed implementation (§6.6 step 4 heals the newest segment in place).

ourios-wal is a single Rust crate exposing one struct Wal whose public API is:

impl Wal {
    pub fn open(config: WalConfig) -> Result<Self, OpenError>;
    pub fn append(&mut self, kind: FrameKind, payload: &[u8])
        -> Result<WalOffset, AppendError>;
    pub fn sync(&mut self) -> Result<WalOffset, SyncError>;
    pub fn checkpoint(&mut self, durable_to: WalOffset)
        -> Result<(), CheckpointError>;
    pub fn housekeeping(&mut self, retain_floor: Option<WalOffset>)
        -> Result<(), HousekeepingError>;
    pub fn last_checkpoint(&self) -> Option<WalOffset>;
    pub fn replay(&mut self, sink: &mut impl FrameSink)
        -> Result<(), RecoveryError>;
    pub fn metrics(&self) -> WalMetrics;
}

pub enum FrameKind {
    OtlpBatch  = 0x01,
    AuditEvent = 0x02,
}

/// Opaque, totally-ordered position of a frame in the
/// WAL. Internally `(segment_uuid: Uuid /* UUIDv7 */,
/// byte_offset_in_segment: u64)`; ordering is
/// lexicographic on the pair, which means **UUIDv7's
/// chronological sort gives global monotonicity** even
/// after housekeeping deletes older segments. A pure
/// `u64` representation would be ambiguous after deletion
/// (no global offset survives reconstruction from the
/// UUID-named files), so the pair is the durable form.
pub struct WalOffset { segment: Uuid, byte: u64 }

pub trait FrameSink {
    fn consume(&mut self, offset: WalOffset, kind: FrameKind,
        payload: &[u8]) -> Result<(), RecoveryError>;
}

Semantics:

• append writes a frame to the current segment and returns the post-append offset. The frame is not yet durable; the caller (the receiver) accumulates appends in a micro-batch and calls sync at the §3.4 batch boundary.
• sync fsyncs the current segment and returns the highest offset that is now durable. The receiver gates its acks on this returning successfully.
• checkpoint records “records ≤ durable_to are on object storage; segments wholly below this offset may be reclaimed.” Called by the Parquet writer’s atomic-publish callback. Reclaim happens on the housekeeping pass, not in-line.
• housekeeping reclaims disk: unlinks every segment whose highest frame offset is ≤ the checkpoint and, when retain_floor is Some(floor), ≤ floor (§6.7). The caller (the ingester) runs it periodically — every wal_housekeeping_secs — passing the latest durable miner snapshot’s high-water mark as the floor; None means no snapshot consumer exists and the checkpoint alone governs.
• last_checkpoint exposes the CHECKPOINT sidecar’s offset (None pre-first-checkpoint). The recovery driver reads it once at startup as the Parquet-side suppression horizon (§6.6).
• replay is recovery: walk every surviving segment in chronological order, hand each well-formed frame — including frames at or below the checkpoint that a straddling or floor-retained segment holds — to sink along with its WalOffset. The offset lets the recovery driver suppress per consumer (Parquet above the checkpoint, miner above its snapshot mark — §6.6); replay itself filters nothing, because an in-replay skip would make a lagging snapshot’s retained frames undeliverable. Used by the ingester at startup before opening network listeners. Returns when the scan completes; the caller then begins serving live traffic.

The crate ships no executable. The receiver and the recovery driver both live in ourios-ingester (RFC 0003).

6.2 Segment layout

Segments live under <wal_root>/<UUIDv7>.wal, where wal_root is the local-disk path configured by the operator. UUIDv7 is the same sortable-by-creation identifier RFC 0005 §3.4 already uses for Parquet files; listing the directory in sorted order yields chronological order, which the recovery scan relies on.

A segment is append-only: the writer holds it open until rotation; readers (recovery) open snapshots read-only — except the §6.6 step 4 heal path, which reopens the newest segment writable to ftruncate(2) a torn tail back to the last valid frame boundary, then closes and reopens read-only before any further reads. The writable window is narrow (one ftruncate + fdatasync + parent-dir fsync) and only ever the newest segment. The segment file format is a header followed by zero or more frames:

| segment-header (24 B)            |
| frame-0 | frame-1 | … | frame-N  |

6.2.1 Segment header

| magic: 4 B = b"OWAL"             |
| version: u16 = 1                 |
| flags: u16 = 0 (reserved)        |
| segment-uuid: 16 B (UUIDv7)      |  → 24 B total

The magic + version pair lets the recovery scanner reject foreign files (e.g. a sibling *.lock) early and pins the format version for future migrations. segment-uuid is the same UUID that appears in the filename; carrying it inside the file too means a mv that mangles the name doesn’t make the file unreadable.

6.2.2 Frame format

| len:   u32_le   (payload length, excluding header + CRC)              |
| kind:  u8       (0x01 = OtlpBatch, 0x02 = AuditEvent; reserved >0x02) |
| _pad:  [u8; 3]  (reserved, MUST be zero, validated on read)           |
| crc32: u32_le   (CRC32-C over kind || pad || payload, Castagnoli)     |
| payload: [u8; len]                                                    |

Per-frame header = 12 B (4 + 1 + 3 + 4). len MUST be ≤ a configured maximum (default MAX_FRAME_BYTES = 16 MiB); larger appends are rejected at Wal::append time before any bytes are written, so a malformed call can’t grow the segment past the cap. Unknown kind values surface as a structured corruption error per RFC0008.5 — the reserved range is how the format admits future frame kinds without a version bump.

CRC choice is CRC32-C (Castagnoli, polynomial 0x1EDC6F41) — the same polynomial the Parquet writer already uses on row groups, so we share one implementation and one SIMD-acceleration path on modern x86 / aarch64. The CRC covers kind || _pad || payload (not the len header, which is implicitly validated by the read returning the right number of bytes; not its own bytes). This is the same shape Kafka uses for record-batch CRCs.

6.2.3 Payload encoding

Payload encoding is keyed on kind:

• FrameKind::OtlpBatch: payload is the OTLP ExportLogsServiceRequest protobuf bytes the receiver decoded — verbatim, no re-encoding. Replay is “re-receive”: the recovery scanner hands the bytes back through the same decoder + tenant fan-out the live request takes. No special replay code path; less surface for the two paths to drift. The WAL’s own format does not embed Ourios’s TenantId; replay rederives it from Resource.attributes per RFC 0003 §6.3.
• FrameKind::AuditEvent: payload is one serialised ourios_core::audit::AuditEvent. The exact serde format is a §9 open question — AuditEvent has no serde derives today (Debug, Clone, PartialEq, Eq only), and the encoder lands alongside them. Strong candidates are bincode (compact, fast, Rust-native) and serde_json (human-debuggable, already in the dep tree). The choice doesn’t affect this RFC’s frame layout; it lives entirely inside the kind = 0x02 payload bytes.

A future per-tenant WAL split would be additive (a new segment-selection layer above the same frame format), as would future frame kinds (the reserved kind range admits them without a version bump).

6.3 fsync policy

Wal::sync calls fdatasync(2) on the current segment file descriptor (or platform equivalent — FlushFileBuffers on Windows; fcntl(F_FULLFSYNC) on macOS when the operator opts into “full” durability). On a sync that follows a segment rotation (a new segment was opened since the last sync, or an older segment was unlinked by housekeeping), it also calls fsync(2) on the parent directory file descriptor — fdatasync is undefined behaviour on directories under POSIX, and fdatasync on the segment file alone does not flush the directory metadata that makes the new entry (or the unlink) durable. The two calls are distinct: fdatasync for the file’s payload + size, fsync for the directory’s entry. The implementation does not call fsync per append; the receiver is responsible for batching appends across the configured window and calling sync once per batch.

Two knobs are exposed at this layer (per CLAUDE.md §3.4 / H3); both are classified in §6.9’s WAL-tunables table:

• wal_batch_window_ms (default 100): the receiver-side upper bound on time from first append to corresponding sync. The receiver implements this; the WAL itself is policy-free at this layer.
• wal_segment_size_bytes (default 128 MiB): segment rotation cap (§6.5). The receiver’s sync MUST also fire whenever this cap is reached on the current segment, even if wal_batch_window_ms hasn’t elapsed — the “or segment fills, whichever first” clause.

6.4 Audit-event ordering barrier

RFC 0001 §6.4 carries the normative requirement (“WAL durability ordering of audit events”) this RFC has to satisfy: a data record carrying template_version = V MUST NOT become durable before every audit event justifying the leaf’s progression to V is durable. Crash recovery may observe [event_1, …, event_k, data_record] or any prefix thereof, but never a data record without the events that caused its version stamp. Without this, replay bumps template_version fewer times than the in-memory leaf did and surviving data records reference a version the audit stream cannot substantiate.

The WAL itself does not enforce the barrier — it provides the primitives (append, sync, the durable checkpoint of §6.7) that the consumer composes into the discipline. The consumer (the miner-side writer in ourios-ingester) is required to follow this sequence for every data record R that emits zero or more audit events E₁, …, Eₖ:

1. Append all audit-event frames for R first: wal.append(AuditEvent, encode(E₁)) … wal.append(AuditEvent, encode(Eₖ)).
2. Track the WAL offset returned by the last audit-event append as R’s required_audit_offset — the audit-WAL position that must be durable before R can be published. If R emits zero audit events (k=0), it has no required_audit_offset and does not contribute to the row group’s max_required_audit_offset in step 3 (max over the empty set is MIN, so the gate in step 4 is trivially satisfied for an all-zero-audit row group — common steady-state).
3. The row group accumulating R’s MinedRecord carries a running max_required_audit_offset across its contained records (the maximum of every record’s required_audit_offset).
4. The Parquet writer’s atomic-publish gate compares the row group’s max_required_audit_offset against the WAL’s last_synced_offset (the largest offset for which Wal::sync has returned). The publish proceeds only when max_required_audit_offset ≤ last_synced_offset; otherwise the row group is held until the next Wal::sync advances last_synced_offset past it.

The comparison is WAL offset vs WAL offset — same units. An earlier draft compared template_version (a small per-leaf integer) against an audit_durable_to position; that was a unit mismatch an implementer could honour incorrectly. The barrier is “is the durable WAL position at-or-past what this row group needs,” not “is the version label high enough.”

This composes with the OtlpBatch frames the receiver appends pre-ack: data records reach the miner via post-ack mining of an already-fsync’d OtlpBatch frame, so the data record’s ingest is already durable; the §6.4 barrier governs the second-stage publish of its post-mining representation to data Parquet. The §3.4 invariant (no ack without WAL fsync) and the §6.4 invariant (no data publish without audit-event WAL fsync) operate on different boundaries and don’t conflict.

Because audit events are emitted in the miner’s hot path, the per-record cost of “step 1 + step 3” is one append call (cheap) plus a shared fsync amortised across many records via the §6.3 batched window. The barrier therefore does not add a per-record fsync.

6.5 Segment rotation

A rotation closes the current segment, fsyncs it (the last fsync that segment ever receives), opens a new segment with a fresh UUIDv7, writes the new segment’s header, and from that point all subsequent appends land in the new segment. The critical-path cost of a rotation is one extra fsync of the parent directory so the new segment’s directory entry is durable before any frame lands in it.

Two triggers:

• Size cap: the current segment’s file size (header + appended frames + the frame about to be written) exceeds wal_segment_size_bytes. Computed before the write so the rotation happens before the next frame’s bytes land.
• Time cap: the current segment’s age (since its header was written) exceeds wal_segment_age_secs (default 600 = 10 min). Bounds the recovery window — a torn-write worst case touches at most one segment-age window of data.

Rotations are silent (no audit event in the steady state). A rotation that fails fsync is a hard error: the receiver must refuse further appends until an operator intervenes, because continuing would risk a frame landing in a segment whose directory entry is not durable.

6.6 Crash recovery

Wal::replay is invoked at startup, before the receiver opens its network listeners:

1. Read <wal_root>/CHECKPOINT (per §6.7) into cp: Option<WalOffset>. A present sidecar yields Some(parsed); an absent sidecar yields None (first-run / pre-checkpoint). A sidecar that is present but invalid — wrong magic, unknown version, non-zero flags, or a size other than 32 B — is a structured corruption error that aborts recovery (the same posture as a corrupt closed segment): silently treating it as None would drop the Parquet suppression horizon and re-feed every already-published record to the data side. The operator restores or removes the sidecar knowingly; removal is an explicit acceptance of at-least-once re-publish. cp = 0 would be a synthetic offset undefined for the (segment_uuid, byte) pair, so absence is modelled as an option rather than a zero value. cp is exposed to the recovery driver as Wal::last_checkpoint() (§6.1) — it is the driver’s Parquet-side suppression horizon, not a delivery filter inside replay (2026-06-12 amendment; see step 3).
2. List every *.wal file under wal_root. Sort lexicographically (= chronologically, per UUIDv7). The last segment in this order is the newest (the one that was open for appends at crash time); every other segment is older / closed (its rotation fsync completed before the next segment began).
3. For each segment in order:
   1. Open read-only, verify the header (magic + version).
   2. Walk frames left-to-right. For each frame:
      • Read the 12 B frame header (len + kind + _pad
        • crc32). If EOF here:
        • Newest segment: clean-truncated tail; stop scanning this segment (RFC0008.4).
        • Older segment: structured corruption error — a closed segment cannot legitimately have a torn tail (rotation fsync’d it before the next segment started), so EOF mid-header is corruption. Emit audit event naming segment + offset; stop scanning all segments (no records past a corrupt point in the log are safe to replay, because the high-water-mark logic depends on contiguous ordering).
      • If len > MAX_FRAME_BYTES, kind is unknown (>0x02), or _pad is non-zero: structured corruption error → same all-segments-stop path.
      • Read len payload bytes. Short read here is the same newest-vs-older fork: a partial payload on the newest segment is RFC0008.4 clean truncation (rotate-pending-fsync at crash time); on any older segment it’s RFC0008.5 corruption (a closed segment with a partial payload means the segment’s final fsync was lost — file-system corruption).
      • Recompute CRC32-C over kind || _pad || payload. Mismatch → corruption (all-segments-stop).
      • Hand (offset, kind, payload) to sink.consume(offset, kind, payload) — every well-formed surviving frame is delivered, including frames at or below cp that a straddling segment retains (2026-06-12 amendment; the pre-amendment design skipped them inside replay, which made the §6.7 retain floor useless — a lagging snapshot’s (S, cp] frames were kept on disk but never deliverable). Suppression is per consumer, in the driver: the Parquet path consumes only frames above cp (frames ≤ cp are already durably published — re-feeding them would duplicate on the data side, since replay is at-least-once and dedup is out of scope), and the miner consumes only frames above its restored snapshot’s high-water mark S (frames ≤ S are already folded into the snapshot — re-feeding them would double-apply, RFC 0001 §6.9 v2). The two horizons are independent and either ordering of cp and S is handled by the same rule: with S ≥ cp the miner consumes a suffix of what Parquet consumes; with S < cp (lagging snapshot) the miner additionally consumes the retained (S, cp] frames that Parquet suppresses. The sink is the recovery driver in ourios-ingester; for OtlpBatch it decodes the bytes as ExportLogsServiceRequest and runs the same tenant-fan-out + miner-ingest pipeline the live receiver does (gated per consumer as above); for AuditEvent it deserialises and reinjects into the audit-event Parquet writer’s queue, gated on the Parquet horizon.
4. Heal the newest segment. If the newest segment’s scan stopped on a torn tail (RFC0008.4 clean truncation), ftruncate(2) the segment file to the last valid frame boundary recorded during the scan, then fdatasync the segment file and fsync the parent directory. Without this step the torn bytes persist and a second crash after a new segment opens would see them on a now-older segment, where RFC0008.5 corruption handling would reject them — contradicting RFC0008.4’s “next append starts at the frame boundary preceding the torn write.” Truncation runs only on the newest segment and only on the torn-tail path; a clean-tail (EOF aligned on a frame boundary) needs no work.
5. After the last segment’s last frame, append-mode resumes on a new segment.

Recovery is single-threaded and synchronous. The operator-visible cost is “ingester is unavailable for the duration of recovery.” RFC0008.3 constrains recovery to O(N) wall time on N segments; the per-frame work is dominated by the miner’s tokenize cost, which is already corpus-bench- governed.

Replay is at-least-once: a batch that was WAL-fsync’d but not yet acked at crash time will be replayed on restart and re-sent through the pipeline. The client may also retry on its end. Dedup is intentionally not in scope here — RFC 0003 §9 carries it as an open question for a future amendment (likely an idempotency key on the batch envelope, persisted in the frame and checked at replay time).

6.7 Checkpoint-driven truncation

The WAL does not truncate itself based on age, count, or total bytes. It truncates only when the Parquet writer explicitly signals that records up to some WAL offset are durably on object storage:

parquet_writer.on_atomic_publish(|durable_to: WalOffset| {
    wal.checkpoint(durable_to);
});

checkpoint persists the offset to a small sidecar file — <wal_root>/CHECKPOINT — atomically (write to CHECKPOINT.tmp, fsync, rename, fsync parent dir). The file format is fixed: 4 B magic b"OWCK", 2 B version = 1, 2 B flags (reserved, zero), 16 B segment UUIDv7, 8 B little-endian byte-in-segment — 32 B total, matching the (segment, byte) WalOffset pair (§6.1). Storing the segment UUIDv7 rather than a synthetic global counter means the checkpoint survives housekeeping deletion of older segments: a deleted segment’s UUID is below any surviving segment’s UUID by definition of the rotation order, so “segments wholly below cp” is well-defined on the remaining files alone. Durability is required, not advisory: if the process crashes after checkpoint(X) but before the housekeeping pass unlinks segments wholly below X, the recovery driver must still know X so its Parquet path suppresses records already published (replay delivers every surviving frame per §6.6; replay is at-least-once and dedup is explicitly out of scope, so the driver’s suppression is the only thing standing between a surviving frame ≤ X and a data-side duplicate). On startup, the CHECKPOINT sidecar is read into Option<WalOffset> per §6.6 step 1 — present sidecar → Some(parsed); absent sidecar → None (first-run / pre-checkpoint) — and exposed to the recovery driver as last_checkpoint. When Some(offset), the driver’s Parquet path suppresses every frame at or below the offset; when None, no Parquet-side suppression applies (replay itself always delivers every surviving frame — §6.6, 2026-06-12 amendment). cp = 0 would be a synthetic offset undefined for the (segment_uuid, byte) pair, so the option semantics is the only well-defined absence representation.

A periodic housekeeping pass (every wal_housekeeping_secs, default 60; the timer lives in the ingester, the pass is the explicit Wal::housekeeping(retain_floor) API of §6.1) walks the segment directory and unlinks any segment whose highest frame offset is ≤ the checkpoint mark and, when a retain floor is supplied, ≤ the floor — i.e. truncation reclaims only segments wholly below min(checkpoint, floor). Segments that straddle either mark are kept until a later pass advances past them (no per-frame deletion; we delete whole segments).

The retain floor (2026-06-12 amendment). The checkpoint guarantees the data side: frames ≤ checkpoint are durably on object storage. It says nothing about the miner state derived from those frames — that lives in the per-tenant snapshot (RFC 0001 §6.9), whose recorded high-water mark S advances on its own cadence (per segment rotation). If a snapshot write fails or lags so that S < checkpoint, truncating up to the checkpoint would destroy the only remaining source of the miner state in (S, checkpoint]: the data is safe in Parquet, but the restored tree would re-mint template_ids for templates first seen in the gap — exactly the hazard #5 template-drift failure. The ingester therefore passes the latest durable snapshot’s high-water mark as retain_floor, and housekeeping never unlinks a frame the snapshot has not captured. None (no snapshot consumer configured) reduces to the checkpoint-only rule. The cost is bounded: the WAL retains at most the segments appended since the last successful snapshot — in the steady state (snapshot per rotation) the floor leads the checkpoint, and the rule is vacuous. The CHECKPOINT sidecar itself always records the true Parquet horizon, never the min — it is the recovery driver’s Parquet-side suppression horizon (read via last_checkpoint, §6.6), and capping it at the floor would re-feed already-published records to Parquet. The floor governs disk reclamation only; the retained (floor, checkpoint] frames stay deliverable through replay precisely so the miner can catch up from them (§6.6).

If the Parquet writer never checkpoints (e.g. a crash before the first row group commits), the WAL grows. The operator- visible signal is wal_disk_bytes rising monotonically; the H3 escalation rule applies. A future RFC may add a fallback “truncate by hard cap” with explicit data loss, but the default is “WAL grows until Parquet catches up” — losing durability silently is worse than running out of disk loudly.

6.8 Metrics

WalMetrics exposes:

• wal_appends_total: u64 — frames appended since startup.
• wal_syncs_total: u64 — fdatasync calls.
• wal_unflushed_bytes: u64 — bytes appended but not yet fsync’d (the H3 detection metric, bounded by RFC0008.9).
• wal_disk_bytes: u64 — sum of file sizes under wal_root (operator dashboard).
• wal_segment_count: u32 — current segment count.
• wal_sync_seconds: histogram — per-sync latency.
• wal_checkpoint_segment: String (UUIDv7 of the segment in the last checkpoint arg) + wal_checkpoint_byte: u64 (byte-in-segment of the same arg). Two fields rather than one because WalOffset is now a (segment, byte) pair (§6.1) and a single u64 would lose the segment identity needed to interpret the value once housekeeping deletes older segments. None checkpoint (pre-first-checkpoint startup) is rendered as the empty string + 0; the ingester logs the transition from “no checkpoint” to “have checkpoint” so operators can tell the difference.
• wal_recovery_seconds: Gauge (set once at startup).
• wal_corrupt_frames_total: u64 — RFC0008.5 surface.

Per docs/roadmap.md §5’s maintainer note (2026-05-19, updated 2026-06-03: “instrument through the OpenTelemetry metrics API (meters) and export via the OTel SDK’s OTLP metric exporter (push), not the legacy prometheus client crate and not a scrape endpoint; any Prometheus compatibility is a downstream collector concern”), these are OTel metrics exported over OTLP; the CLAUDE.md §6.3 “every subsystem exposes Prometheus metrics” line predates that direction and the note flagged it for a follow-up amendment. RFC 0001 §6.8 (2026-06-03 amendment) is the normative reference for the export architecture.

6.9 WAL-tunables classification

Per RFC 0004 §3.1, every operator-visible knob is exactly one of Tunable or Invariant, with a startup- validated range and the §3 invariant it lives inside. Same schema as RFC 0004 §3.2’s miner table:

Per RFC 0004 §3.1’s per-tenant override rule, all Tunables above are process-global at v1 — there is no per-tenant override surface for the WAL because the WAL is a single workspace-wide log (§7.4). A future per-tenant WAL split would inherit RFC 0004’s per-tenant override mechanism.

6.10 Out of scope for this RFC

• Replication. Per CLAUDE.md §3.4. When it lands it joins a separate RFC and operates over the WAL, not instead of it.
• Multi-writer arbitration. The ingester is a single process per node; no file lock dance here. A future multi-process design would need a lockfile + a coordinator, both deferred.
• Snapshotting miner state. Replay rebuilds the miner; if recovery time becomes the operator complaint, a snapshot RFC is the response. §9 carries this.
• Batch dedup on replay. RFC 0003 §9 owns the idempotency-key question; this RFC inherits whatever that RFC pins.
• Distributed transactions across WAL + object storage. Atomic-publish (RFC 0005 §3.7 / RFC0005.2) is what we have; this RFC does not promise stronger ordering.

7. Alternatives considered

7.1 Use sled / redb / fjall / RocksDB

These give a WAL as a side effect of a B-tree / KV store. The ingest path consumes none of the surrounding API: we don’t iterate keys, we don’t query by prefix, we don’t transact across rows. Adopting one buys us a WAL implementation at the cost of a much larger dependency, a non-trivial migration if the upstream goes through a format break (sled has done this multiple times pre-1.0), and a wire format we don’t control. The hand-rolled crate is ~600 lines we can audit; the alternatives are tens of thousands of lines of code we don’t need.

7.2 Use okaywal / a focused-WAL crate

Closer fit (focused, small, hand-roll-equivalent). The remaining objection is that the format is what we care about most — recovery semantics, frame layout, CRC choice, checkpoint contract — and a third-party crate’s format is the same wire-compat boundary as our own. We do not save the design work, only the implementation work. For ~600 lines of careful Rust the dependency-discipline default (CLAUDE.md §10 “When in doubt, …”) is to own it. Worth revisiting if okaywal (or a peer) stabilises on a format that matches §6.2 closely enough that we can adopt them as a drop-in.

7.3 Per-record framing (not per-batch)

Per-record gives finer recovery granularity (“the last K records before crash were lost” vs “the last batch was lost”) and bypasses one decoder pass at replay (records are already split). Rejected because:

• Per-record framing forces the receiver to re-encode each record from the decoded OtlpLogRecord back into a durable form, doubling the encode work on the hot path.
• Batches are the ack boundary already (RFC 0003 §6.5); a per-record WAL would have to track batch-boundary metadata anyway to know which records to ack together.
• Replay-as-receive is a strong simplicity win: one decoder, one fan-out, one mining pipeline. Per-record framing forks the code.

7.4 Per-tenant segments

A segment per (tenant_id, time-bucket) would let recovery parallelise by tenant and would shrink the truncation granularity (a slow-flushing tenant doesn’t pin the whole WAL). Rejected for v1 because tenant derivation (RFC 0003 §6.3) happens inside the batch — the WAL would need to peek inside ExportLogsServiceRequest to route, which re-introduces decode work on the critical path. The shared log keeps the WAL stateless about tenants; per-tenant segmentation is a future optimisation that the §6.2 format admits additively.

7.5 fsync per append

The simplest possible policy: every append fsyncs. Rejected because the latency cost is operator-hostile on the steady- state path (a single batch with 100 frames → 100 fsyncs, each ~10 ms on commodity SSDs). The batched-window approach preserves the §3.4 invariant (no ack before fsync) while amortising the cost across the configured window.

7.6 Direct I/O (O_DIRECT)

Skips the page cache for a known cost (fsync always flushes) in exchange for a known benefit (no kernel read-modify-write on small appends). Rejected for v1: the implementation surface is non-trivial (aligned buffers, sector-aligned writes), the benefit is operator-perceptible only at multi-thousand-batch-per-second sustained rates, and the H3 mitigation already gives us a tunable knob. Worth revisiting if production deployment shows fsync amplification as the dominant ack-latency contributor.

7.7 OTLP-protobuf vs bincode / flatbuffers

bincode would be slightly faster to encode and would shed the proto3 overhead, but the receiver has the protobuf bytes in hand for free (the wire decoded into them) — using them verbatim is zero work, while re-encoding into bincode would be a second pass. flatbuffers would require schema duplication we don’t currently maintain. The OTLP wire form also keeps the WAL self-describing to anyone with the opentelemetry-proto headers; a custom format would be ours-only.

8. Testing strategy

Per CLAUDE.md §6.2 (docs/verification.md §2), each §5 scenario maps to a named test:

• RFC0008.1 — integration test in crates/ourios-wal/tests/wal_before_ack.rs. Drives a fake receiver against a real Wal; asserts the sync-returned offset is the gate.

• RFC0008.2 — CI crash-recovery test (the H3 normative requirement). A test harness forks a child that appends N frames; the parent sends SIGKILL between append and sync and (in a second arm) between sync and the simulated ack. The assertion has two halves:

  1. No fsync’d frame is lost. Every frame the child fsync’d before the kill is recovered on restart.
  2. Any un-fsync’d frame is handled safely. Process death does not discard dirty page-cache data, so a complete-but-unsynced frame may still be on disk after restart (the kernel flushed it post-mortem). Recovery MUST sort it into one of the three §5 / §6.6 buckets: replayed as a normal frame (complete + CRC valid → legitimate, just unacked at the client); surfaced via the newest-segment torn-tail truncate (partial header / partial payload only — §6.6 step 4); or surfaced as RFC0008.5 corruption (complete frame with CRC mismatch, even on the newest segment — CRC mismatch is never truncation). The test does not assert “exactly the fsync’d frames” — that would be stronger than what SIGKILL semantics admit.

  Runs on every PR; failure blocks merge.

• RFC0008.3 — criterion benchmark in crates/ourios-wal/benches/recovery.rs. Generates 1 / 4 / 16 segments of representative size, measures recovery wall time, asserts O(N) growth within a tolerance.

• RFC0008.4 — property test in crates/ourios-wal/tests/torn_writes.rs. Generates a segment, truncates at a random offset inside the last frame, asserts replay succeeds and the recovered frame count equals the count before the torn frame.

• RFC0008.5 — property test (crates/ourios-wal/tests/corruption.rs). Five arms, one per §5 RFC0008.5 sub-case, matching the §6.6 corruption branches:

  1. Payload bit-flip (CRC mismatch) on a closed segment — flip a random bit in a random frame, assert replay emits the structured corruption error and stops scanning all segments.
  2. Unknown kind — generate a segment containing a frame whose kind byte is anything other than 0x01 or 0x02 (i.e. inside §6.2.2’s reserved 0x03..=0xFF range — bytes “reserved for future kinds” are rejected today, because no future kind has been added); assert structured corruption.
  3. Non-zero _pad — generate a frame whose 3 B reserved _pad carries a non-zero byte; assert structured corruption (the §6.2.2 contract says _pad MUST be zero and is validated on read).
  4. Oversize len — generate a segment whose frame header declares len > MAX_FRAME_BYTES; assert structured corruption (the read must reject the header before attempting to read len bytes that could exceed the segment size).
  5. Torn header / payload on a closed segment — truncate a closed segment mid-header or mid-payload; assert RFC0008.5 corruption, not RFC0008.4 clean- truncation (the newest-vs-older pin §6.6 introduced).

  Each arm asserts (a) the structured error names the segment UUIDv7 + byte offset and (c) recovery stops scanning all segments (no records past the corrupt point are replayed). The original (b) — the WalRecoveryCorruption audit event — is deferred per the 2026-06-13 §5 amendment (system-scoped audit, §9); wal_corrupt_frames_total carries the operator-visible signal in the interim.

• RFC0008.6 — integration test (crates/ourios-wal/tests/rotation.rs). Drives appends past the size cap and time cap; asserts no frame is dropped or duplicated and the segment-file count grows by one.

• RFC0008.7 — integration test (crates/ourios-wal/tests/checkpoint.rs). Four arms:

  1. Normal-flow truncation — checkpoint(X) followed by a housekeeping pass unlinks segments wholly below X and keeps segments straddling it.
  2. Crash between checkpoint(X) and housekeeping — fork a child that calls checkpoint(X), SIGKILL the child before housekeeping runs; restart, assert the CHECKPOINT sidecar survives the crash: last_checkpoint() returns Some(X), so the driver’s Parquet-side suppression of records ≤ X holds (they aren’t re-fed and duplicated on the data side), while replay still delivers them with their offsets. Without this arm a passing test could still hide a non-durable checkpoint (the §6.7 sidecar requirement).
  3. Surviving-segments offset reconstruction — checkpoint(X) advances past several older segments, housekeeping deletes them, then a fresh Wal::open plus a new checkpoint(Y > X) proceeds against the surviving UUID-named files without needing a global offset counter to be reconstructed — verifying the §6.1 WalOffset = (segment, byte) representation is enough.
  4. Retain floor — with checkpoint(X) and a floor S < X, housekeeping(Some(S)) keeps every segment holding frames in (S, X]; advancing the floor to S' ≥ X on a later pass reclaims them.

• RFC0008.8 — integration test that drives the wal_batch_window_ms knob at {10, 100, 1000} ms under a paused virtual clock and asserts ack latency equals the configured window exactly (deterministic — the window dominates, not per-record fsync).

• RFC0008.9 — property test on the metric: wal_unflushed_bytes is monitored over a randomized arrival pattern and asserted bounded by 2 × wal_segment_size_bytes.

• RFC0008.10 — integration test in ourios-ingester / ourios-server (the driver lives there per §6.1, like the RFC 0001 criteria hosted in ourios-querier). Builds a WAL

  • snapshot fixture with S ≥ X, starts the server, and asserts: recovery completes before the listeners accept; per-consumer delivery counts (Parquet sees all frames

  X, the miner only frames > S); and tree-state equality against a from-scratch control. The snapshot-cadence arm drives appends across a rotation and asserts a snapshot artefact appears recording the rotation-point high-water mark.

The crash-recovery test (RFC0008.2) is the single most important test in this RFC and is also the H3 CI gate. It runs on every PR regardless of which files changed — H3’s “any failure is critical” applies.

9. Open questions

• System-scoped audit for WalRecoveryCorruption (deferred from RFC0008.5, 2026-06-13). RFC0008.5’s durable forensic record — a WalRecoveryCorruption { segment, offset, reason } audit event — was deferred because the RFC 0005 audit stream is tenant-partitioned (CLAUDE.md §3.7) and WAL corruption is a system event with no tenant. The follow-up needs a system-scoped audit partition (a reserved tenant id, or a sibling audit/system/ tree) decided in an RFC 0005 amendment; once it exists, the recovery driver appends the event post-scan (recovery is single-threaded, so the queue drains after the halt). Until then the structured RecoveryError halt + wal_corrupt_frames_total are the operator-facing signal. The same system-scoped partition would host other infrastructure events (rotation-fsync failure, checkpoint-write failure) that today only surface as metrics + the quiesce/error response.
• AuditEvent serde format. AuditEvent has no serde derives today (Debug, Clone, PartialEq, Eq only), so the FrameKind::AuditEvent payload encoding is intentionally deferred to the implementation PR — this RFC at specified pins the frame layout and the ordering contract (§6.4), not the inner encoding. The encoder lands alongside the serde derives in PR-M2; the candidates are bincode (compact, fast, Rust-native — picks up cross-version compatibility concerns later if the struct evolves) and serde_json (human-debuggable, already in the dep tree, larger on disk). Either lives entirely inside the kind = 0x02 payload bytes and doesn’t affect the frame layout this RFC pins. The choice should be consistent with whatever encoding the audit-event Parquet writer (RFC 0005 §3.7) uses on the read path.
• Idempotency key on the batch envelope. RFC 0003 §9 carries the at-least-once duplicate problem; the WAL frame can embed a key for replay-time dedup. Sequencing: whichever RFC moves first proposes the key shape, the other amends to consume it.
• macOS F_FULLFSYNC policy. macOS fsync doesn’t flush the on-disk cache without F_FULLFSYNC. Default to the stronger primitive (slower, correct) or the weaker one (faster, dependent on power-loss assumptions)? Most Ourios deployments are Linux, so the default doesn’t bite the common case, but the macOS dev-laptop bench path benefits from the weaker default. Lean toward an explicit wal_macos_full_fsync = true knob with a documented trade-off.
• Recovery-time snapshot of miner state. If replay on a hot-template multi-tenant corpus exceeds the operator’s restart-window budget, a snapshot of the miner’s Drain trees becomes worth specifying. Currently deferred with no signal that it’s needed. RESOLVED 2026-06-12: specified in RFC 0001 §6.9 (format + v1 landed 2026-06-10; the v2 restore-and-resume amendment is the counterpart of this RFC’s same-day §6.1/§6.7 amendment — offset-carrying sink, retain floor, RFC0008.10 driver).
• Cross-segment frame straddling. The §6.2 format forbids it (a frame lives in exactly one segment); the rotation rule pre-checks size before appending. Confirm the receiver-side append loop respects this; one test in §8 already covers it but the open question is whether the knob should be tunable (very large batches near the rotation point) or fixed (always rotate ahead of the oversize append).
• What does the receiver do when sync returns error? Refuse acks for that batch is the obvious answer; less obvious is whether the receiver tears down its network listener (force the orchestrator to restart it) or holds incoming requests until the error clears. RFC 0003 §9 is the natural owner of this once the WAL contract is specified.
• Multi-WAL on a single node. Could a single ingester run two Wals (e.g. one per fast-tier disk for IO parallelism)? §6 currently pins a single WAL per process; a multi-WAL design would need a coordinator and a cross-WAL ordering rule. Deferred.

10. References

• CLAUDE.md §3.4 WAL-before-ack — the load-bearing invariant.
• CLAUDE.md §3.6 Object storage is the source of truth — the WAL is local-disk cache + durability, not the persistent record.
• CLAUDE.md §10 When in doubt, … — informs the hand-roll vs adopt-a-crate choice in §4.2 / §7.1.
• docs/hazards.md H3 WAL durability vs. latency — the mitigation, detection, and escalation language this RFC promises to honour.
• RFC 0003 §6.5 WAL-before-ack sequencing — the receiver contract this RFC is the other side of. RFC 0003 cannot progress past drafted until this RFC reaches specified (because RFC 0003’s §5 scenarios cite the WAL).
• RFC 0005 §3.4 Partition layout on disk — provides the UUIDv7 convention §6.2 reuses.
• RFC 0005 §3.7 / RFC0005.2 Atomic publish — the per-row-group publish primitive §6.4 (audit-event ordering barrier) and §6.7 (checkpoint-driven truncation) compose against.
• RFC 0005 §3.3 — the canonical-JSON encoding for attributes / structured bodies, which the replay pipeline consumes downstream of the receiver.
• RFC 0001 §6.1 — OtlpLogRecord shape, the in-memory form the recovery sink fans out to.
• RFC 0001 §6.4 / §6.7 — AuditEvent schema and the audit-stream-through-WAL contract this RFC implements via FrameKind::AuditEvent.
• RFC 0005 §3.7 — audit-event Parquet schema; explicitly defers the audit-WAL contract to this RFC (“a contract that lands with the post-MVP ourios-wal crate; until then audit-event durability is in-memory and the corpus bench accepts that”).
• docs/roadmap.md §5 — the maintainer note (2026-05-19, updated 2026-06-03) pinning OTel metrics (via meters) + OTLP export (not direct Prometheus, not a scrape endpoint) as the metrics surface; the §6.8 instrumentation cites it.
• PostgreSQL WAL design notes — informed the segment-rotation + checkpoint-truncation pattern.
• LMDB B+tree on a memory-mapped file — informed the “tiny, owned, audited” approach in §4.2.
• Kafka log segment format — the per-frame length-prefixed CRC-checked layout in §6.2.2 is the conventional one and follows Kafka closely.