rfc: 0052 title: WAL reclamation and quiesce recovery status: green author: Jens Holdgaard Pedersen jens@holdgaard.org drafting-assistance: Claude created: 2026-09-12 supersedes: — superseded-by: —

RFC 0052 — WAL reclamation and quiesce recovery

Status: green (2026-09-30, maintainer-approved flip). Every live §5 scenario passes. RFC0052.17’s two PUBLISHED legs moved to RFC 0055 as RFC0055.5 before the flip, because RFC 0055 (publication frontiers) owns the PUBLISHED writer and format. Their stub stays #[ignore]d under that criterion, as .6, .8 and .9 moved to RFC 0053. The recorded long soak run §6 requires before validated is still to come. The history of the red stage follows.

(red — test stubs existed and failed (docs/rfcs/README.md §Lifecycle).) Every live §5 scenario was stubbed as one or more #[ignore]d todo! stubs, per leg where §6 separates the legs, in ourios-wal/tests/it/rfc0052_* (.2, .4, .5, .11, .12, .13, .16, .17), ourios-ingester/tests/it/rfc0052_* (.1, .10, .13’s startup leg, .14, .15), ourios-ingester/tests/rfc0052_7_telemetry/ (.7, its own binary per RFC0028.2) and ourios-bench/tests/rfc0052_3_bounded_growth.rs (.3). Slices A–E have landed and discharged their stubs, .7 included, and slice F’s first half (F1) has discharged .10: a SIGKILL of receiver_reclaim_crash_fixture with the barrier and housekeeping on a 100 ms cadence loses no acknowledged record, and recovery now gates the replayed records on max(X, S) and the regenerated audit events on X (§3.7), withholding and counting the rest. The barrier still retains a cut whose snapshot write failed (RFC0052.7 pins that), so the S < X leg reconstructs that state on disk rather than reaching it through a cut. Slice F2 has discharged .3 on the reclamation soak harness (ourios-bench/src/soak/reclaim.rs), which runs the receiver’s barrier and housekeeping ticks on a stepped synthetic clock: a capacity-balanced run over three tenants rolls twelve segments under the 300 s / 60 s cadences and never holds more than the bound its config derives (four segments), and an idle node reclaims every closed segment one barrier_secs plus one housekeeping_secs after its last append, and its last segment within segment_age_secs more. The recorded long soak run §6 requires before validated is still to come. The only stub still ignored is .17’s PUBLISHED row. That row is recorded as partial at green and handed to RFC 0055, as RFC 0041 handed on its deferred rows: this RFC turns green on the other rows once slice F lands, and RFC 0055’s writer discharges the row. Implementation proceeds in six green slices, each un-ignoring the stubs it discharges. Where a criterion’s legs span slices, the mapping below is by leg and each stub’s #[ignore] reason names its slice: A reclaim record and sidecars (§3.2 RECLAIM, CHECKPOINT v2, SEGMENT_VERSION 2, open-time reconciliation → .16, .11’s open-time leg, .17’s format, sidecar-matrix and geometry legs: the corrupt record, the deleted record on a post-RFC root, the overlong replayed tenant, the both-sidecars-missing rows, the geometry rebuild, the at-open allocation failure, the fresh-open ordering, the torn-slot fallback and the checkpoint_armed / checkpoint_seen rows); B housekeeping (ledger, RetainFloor, capped passes, prepare/commit split → .2, .12, .13’s floor legs, .17’s pass legs and every leg that reads a snapshot, since SnapshotHorizons arrives at housekeeping_prepare: the halt-or-pin rows, the empty-record pin, entry monotonicity, the record-then-unlink crash, the legacy-root migration window (whose stale-gap leg compares a tenant’s oldest surviving frame with its last recorded horizon), the consumer-mode rows, the migration-window skip, failed or uncertain unlink, failed record write). .17’s PUBLISHED seeding and slot-id row is not slice B’s: both its legs read the PUBLISHED sidecar, and while §3.2 defines the two published_seeded_* header bits as RFC 0053’s amendment to this header, the file’s writer and format are RFC 0055’s (publication frontiers, drafted). The stub’s earlier reason misattributed it to RFC 0053; it stays ignored, named for RFC 0055, and the slice that lands that writer discharges it; C rotation (temporary name, bounded retry, terminal state, the hold/794-wedged-classification reintroduction → .4, .5, .15, .11’s post-RFC rotation leg, .17’s legacy-root rotation leg); D barrier (guard-at-submit, publisher thread carrying detached partitions only, epoch latch, ingest exclusion → .1’s cut legs, .14, .13’s startup leg); E timer and telemetry (§3.5 instruments and events → .7, .1’s cadence-tick panic leg); F crash and soak (.10 on the rfc0014_5 fixture, .3 on the extended soak harness). A before B; B before D and E; F last. Snapshot v2 (#877) sits outside the six slices and has landed as its own breaking change: §3.1’s per-tenant folded horizon in wal_high_water (each ingest turn and each replayed frame folds its tenant’s horizon, and the cut, the post-recovery and shutdown writes and the ledger carry it per tenant) and SNAPSHOT_VERSION 1 → 2, a version-1 artefact taking the unknown-version discard path. The same landing restates recovery’s stale-gap check on the RECLAIM record (§3.2): an absent horizon segment is explained when the tenant’s reclaimed-through is at or above S. The §8 amendment of RFC 0001 §6.9’s global high-water wording is live from that landing. Stage 1 of two. Motivated by a production incident (issue #791) and the defects found tracing it (#791, #793). Amends RFC 0008 §6.5 and §6.7 with the policy those sections left to a caller that was never written, amends RFC 0001 §6.9 (per-tenant own-frame snapshot horizons and the tenant-aware retain rule supersede its global high-water wording; §8), amends RFC 0018 §3.2 (whose transient class lists “post-rotation quiesce”, which #791 disproved), and exports the WAL telemetry the incident showed is missing. Touches CLAUDE.md §3.4 throughout, which is why it is an RFC and not patches.

Staging. An earlier draft also carried explicit backpressure and unwind safety (#796). Six review rounds found most defects at the seams between the four parts, and the maintainer directed a split: this RFC is reclamation and rotation recovery — the pair that today guarantees every node eventually wedges — and RFC 0053 is backpressure and unwind, which depend on this one and land after it. Section numbers are kept stable so the review history still resolves; §3.4 and §3.6 are pointers.

1. Summary

The WAL has never reclaimed a segment in any deployment: checkpoint and housekeeping are called only from tests, so wal_housekeeping_secs times nothing and the WAL grows monotonically for the life of the volume. When the volume fills, rotation fails and sets a quiesced latch that no code clears, so every subsequent write is refused until the process is restarted — and the refusal carries no log, no metric, and (until #794) no reason. This RFC commits to: advancing the checkpoint at the existing publication barrier and running housekeeping on the existing timer; making a rotation failure recoverable in place under a bounded retry rather than permanent; and exporting the WAL state an operator needs to see either happening. Explicit backpressure and unwind safety, which depend on both, are RFC 0053.

2. Motivation

2.1 The incident

A transient object-store outage wedged a single node for eight hours (#791). Ingest answered 503 with an empty body for every tenant, in 3 ms — a local refusal, not a timeout — and kept doing so for the last eight of those hours, during which the object store was verified healthy. systemctl restart cleared it in 21 seconds. The downstream collector dropped 4,526 journal records and 177 application log records before anyone noticed, and what noticed was an alert on the collector’s queue depth, one hop away. Ourios itself emitted no signal at all.

That is a direct contradiction of the promise §3.4 makes. WAL-before-ack is supposed to mean a flaky object store degrades querying and delays compaction while writes keep landing on local disk. Here a backend blip took writes down and kept them down after the blip ended.

2.2 Two defects, one question

Tracing it turned up defects that are all the same question seen from different sides — when is it safe to declare WAL frames reclaimable, and what happens when we cannot? The two this RFC takes are the pair that guarantees the wedge; a third, #796, is RFC 0053’s.

#793 — the WAL never truncates. Wal::checkpoint (§6.7) and Wal::housekeeping are sound, specified, and tested. They are also dead: every call site is a test. Production reads last_checkpoint() during recovery and never writes one, and housekeeping is documented as a no-op before the first checkpoint, which never comes. The incident node held 1,113 segments totalling 42 MB after five days — all age-rolled (the minimum segment size is 17 MiB), none ever reclaimed. On a long-enough timeline every node fills its volume.

#791 — the quiesce latch is permanent. Wal::rotate sets quiesced = true at four sites, and no code anywhere sets it back. The field doc says “until an operator intervenes” and §6.5 says “operator intervention is required”, but no intervention exists: there is no WAL verb in the CLI and no API to clear the flag. Wal::open is the only reset, so the intervention is, in practice, a restart. ENOSPC from #793 is the most likely way to reach it, but any of the four I/O steps failing is sufficient.

#796 — a publish unwind loses drained batches is the same question from the sink’s side rather than the WAL’s, and is specified in RFC 0053 §2.2. It is mentioned here only because #795’s stopping the sweep on a panic — the behaviour this RFC’s reclamation must coexist with — exists to bound it.

2.3 Why at this layer

Each of these could be patched locally, and two of them were (#794 makes the rejection legible, #795 makes a dead cadence alertable). Neither patch could touch the behaviour, because both decisions turn on the same invariant: CLAUDE.md §3.4 says an acknowledged record must not be lost, and every candidate fix either declares frames reclaimable (checkpoint, truncation) or decides whether to keep accepting them at all (quiesce recovery). Getting either wrong loses acknowledged data. That is the definition of an RFC-level change.

The reclamation half is also not a new design. RFC 0008 §6.7 already specifies the mechanism precisely — monotonic checkpoint, sidecar, min(checkpoint, snapshot_floor) truncation bound, whole segments only, never the current one. What it deliberately left out is the policy: which offset, on what condition, on whose timer. This RFC supplies that, and the answer turns out to be constrained almost to a single option by code that already exists.

3. Proposed design

3.1 Checkpoint at the existing publication barrier

flush_then_snapshot already computes exactly the predicate a checkpoint needs. It returns true only after: in-flight encodes are quiesced, the coordinator’s drained-in-flight publishes have settled (quiesce_publishes), the audit sink has fully flushed, and the record sink reports zero buffered records. Its own comment states the consequence — “every acked record at or below the mark is durably captured”. That is the checkpoint precondition, already proven, already under the miner lock.

So the checkpoint advances behind that barrier and nowhere else. But the barrier must also be reachable without an append. Today it runs from the rotation hook, which fires only after a successful append observes a segment change, and from shutdown. If reclamation depended on that alone, a node that stops receiving traffic would never advance its checkpoint again — no append, no rotation, no barrier — and an idle node is exactly the one whose retained segments have the least reason to exist. RFC 0053’s backpressure sharpens the same gap into a deadlock (a bound that rejects every append can then never be cleared by one), which is why it depends on this timer rather than adding its own. The age sweep is not a substitute: it deliberately takes no snapshot and so establishes nothing about durability.

The barrier therefore also runs on §3.2’s timer, which is append-independent by construction. One predicate, three callers (rotation, shutdown, timer) — and the timer caller carries the same prologue the other two already do: flush_then_snapshot does not quiesce encode submissions itself, its callers do it first, so a timer caller that skipped quiesce_encodes() would stamp across in-flight encodes (the barrier’s own class-1 case) and lose exactly what it is meant to protect. “One predicate, three callers” means the whole sequence, not the inner function.

Calling the prologue is still not sufficient, and this is where the timer differs from the other two callers. Rotation calls quiesce_encodes() while holding the pipeline’s miner lock, and shutdown has already stopped the listeners; each therefore has an exclusion keeping new submissions out between the quiesce and the stamp. A bare timer has neither, so ingest can submit a fresh encode in that window and the barrier stamps across it — the same loss, reached by a narrower race.

And the exclusion is the ingest gate, not the miner lock. That distinction is the whole of it: ingest_bound releases the miner lock before pool.submit(mined) and before advancing last_durable, so a timer holding only that lock can quiesce the pool and then have an already-past-the-lock ingest submit an encode and advance the mark underneath it. The barrier would stamp at a mark whose encode was never quiesced — the same loss again, one layer down.

What actually makes rotation safe is that it runs inside an ingest’s gate turn, and the gate serializes turns in order: “every frame ≤ this seq submits its encodes before any later seq reaches the rotation check”. So the timer must become a participant in that ordering rather than a concurrent holder of a narrower lock.

The existing gate cannot give it one, so this is a change to the pipeline, not a use of it. ingest_gate is a watch counter that append callers advance with their own commit sequence; await_ingest_turn waits for a sequence, and a timer has none to reserve. Inventing a synthetic sequence would interleave with real ones and could stall an ingest behind a barrier that never arrives.

Instead the pipeline gains an explicit barrier exclusion: a lock ingest_bound holds across the span that matters — from the miner work through pool.submit and the last_durable update — and that the timer takes exclusively to capture a cut, and for nothing else. It is strictly wider than the miner lock and strictly narrower than the gate, so it does not change ingest ordering, and it is the smallest thing that closes the window. Rotation’s exclusion needs no change — it already runs inside an ingest’s own span — but its I/O does. Today rotate_for_segment_change runs under the miner lock and the gate and performs the whole of flush_then_snapshot there, PUTs included; under this RFC that span is also the barrier exclusion, so leaving the hook as it is would hold the exclusion across store I/O at every rotation — exactly the stall the next paragraph takes the timer out of, and with §3.3’s retry it would recur on every retried rotation during an outage. So the rotation hook becomes capture-only: inside the turn it quiesces the encodes, takes the mark (prev, the rotation point), drains both sinks into owned batches registered as in flight, serialises the snapshot bytes, and hands that cut to the barrier task, which runs the same store-I/O sequence the timer’s own cuts run (§3.2’s pseudocode from cut_ok on). The handoff is a single pending slot, not a queue, so it is O(1) and never blocks the turn and cannot grow during an outage: the barrier task holds at most one pending cut beside the one it is running, and a rotation capture that arrives while a cut is pending coalesces into it — the new drained batches are appended to the pending cut’s set (they are disjoint by construction, each drain emptying the buffers), the mark advances to the newer prev, the snapshot bytes are replaced per tenant by the newer capture’s (latest wins; a tenant absent from the newer capture keeps the older bytes, which are still cut-consistent at its own horizon), and the epoch becomes the newer capture’s, which is what makes the coalesced cut fail on any guard registered between the two captures. A capture that arrives while nothing is pending fills the slot. The slot bounds metadata, not payload, and that distinction matters during an outage: coalescing appends drained batches, so a slot that keeps absorbing captures accumulates acknowledged records outside the sink’s byte accounting, which is the one place the bound is measured. So coalescing is itself bounded — the pending cut holds at most one cut’s worth of batches, capped by the sink’s ceiling byte figure (ceiling_bytes, the limit the buffers already answer to) — and a capture that would take it past that ceiling does not coalesce at all: it parks every batch it drained back into the sink’s buffers as ready partitions under their accounting, with the same epoch-dated park §3.1 gives every other park, and advances nothing — the pending cut keeps its own earlier mark, its own snapshots and its own epoch. Parking only the excess would break the invariant the whole design rests on: the pending cut’s mark would have moved to the newer capture’s prev while some of the frames at or below it sat in the buffers rather than in the cut’s batches, and run_cut would checkpoint through them. All or nothing per capture keeps “every frame at or below the mark was drained into this cut’s batches” literally true, and costs only that the parked partitions wait for the next cut, which drains them at a mark that does cover them. The bound is deliberately the sink’s own, not a new number: RFC 0053 owns the client-facing limit, and nothing here refuses a client. Cuts are strictly ordered, and a pending cut inherits the running cut’s outcome. The pending cut was captured while the running cut’s batches were still in flight, so its snapshot bytes already fold frames whose only durable copy is the running cut’s batches: if it installed or stamped first — or after the running cut had failed — those frames would be suppressed on replay while sitting requeued in the buffers. So the pending cut never installs or checkpoints before the running cut’s outcome is known, and when the running cut fails (a partition’s flush fails and requeues) the pending cut is invalidated: its snapshot bytes are discarded, its batches — none of them flushed yet, the task being sequential — are merged into the buffers with the requeued ones under the sink lock, its mark is withdrawn, and its epoch is spent; the barrier task then re-captures a fresh cut from the buffers under the exclusion, which covers everything either cut held. The same rule reaches the timer’s own cuts through prior_ok (§3.2): a failed publish is consumed by its requeue, which records the barrier_epoch current at the requeue, and a cut is valid only if no publish registered before its capture requeued at an epoch above the cut’s — a cut captured after the requeue covers the records and is valid, one captured before it is not. At shutdown the receiver joins the barrier task after the running run_cut finishes and the pending slot, if filled, has been run once — a store failure there requeues into the buffers as any cut’s does, and shutdown’s own final flush covers the requeue. The ingest turn does no store I/O; the mark invariant is unchanged, because every cut was captured under the exclusion; and there is one owner of barrier I/O rather than two. The rotation hook and the timer are then the same barrier with two triggers, which is also what lets §3.7’s Journal::rotate fire a rotation from the timer without a second code path.

The cut is captured under the exclusion; the store I/O is not. Under the exclusion the timer stops new submissions and waits for the encode phase of every batch submitted before the cut to finish — not for store I/O. The unit is the batch, not the worker: a batch is encode-complete when each of its records has been appended to the buffers or detached into a registered publish, and the pool already counts exactly that (submit increments the pending count, BatchGuard decrements it when the worker’s loop over the batch ends), so quiesce’s unit stays what it is. Waiting on anything finer is unsound: a worker takes a batch record by record, and emit_concurrent detaches and publishes partitions one at a time, so a quiesce that returned once a worker had registered a publish would leave the rest of that worker’s pre-cut batch unemitted — records at or below the mark that the cut’s drain cannot see and the checkpoint would stamp past. What changes is that the worker no longer performs store I/O: EncodePool::quiesce today waits for a worker even while it is inside a put_blocking, and the sink’s in-flight guard today covers only PublishCoordinator::drain_*, not the publish_owned calls emit_concurrent makes itself, and today there is no publisher to hand anything to: publish_owned and write_ordered are both synchronous on their caller. So this RFC adds one, and its lifecycle is stated here and nowhere else. The publisher is one dedicated thread owned by the PublishCoordinator — the object that already owns both sinks and runs the age sweep’s write_ordered — fed by a bounded queue of detached batches. The worker never blocks on that queue: a bound the worker waited on would put a stuck PUT back in front of the barrier — the queue fills, the enqueue blocks the worker, and quiesce_encodes() waits on that worker while the exclusion is held. So on a full queue the worker parks the detached partition back into the sink’s buffers as a ready partition — still under the sink’s byte accounting, excluded from the size trigger (it is already past it), and taken by the publisher, oldest first, as the queue drains, or by the next drain or flush like any buffered partition — and moves on. A park is a settlement with a date on it, exactly like a requeue: it records the barrier_epoch current when the records re-entered the buffers, and quiesce_publishes().all_ok(cut.epoch) is false for any cut captured before that epoch. Without that, parking would be the one way out of §3.1’s predicate — a pre-cut guard could park its partition after this cut had already drained the buffers, the guard would settle successfully, and the barrier would checkpoint over acknowledged records that exist only in the buffers it no longer holds. Every one of the three park sites takes the rule: the full-queue park here, the publisher’s unwind park, and the closed-channel park. A cut captured after the park drained those records itself and stamps normally. A ready partition is not a plain record partition: it keeps its audit_watermark, carried in the buffer entry beside the records, because parking must not strip the dependency that makes the records publishable. Every consumer of a ready partition — the publisher, an age drain, the barrier’s own flush — honours the same wait-or-write rule §3.1 states below: the events up to that watermark must be durable before the records are written, and a Drained that includes a ready partition carries the maximum watermark over the partitions it took. Parked records are in the buffers, so they are covered by every cut’s drain and need no publish guard; the queue depth and the parked partition count are exported (an UpDownCounter and a gauge, registry names in the one bump) as the mechanism’s only signal — RFC 0053 owns the client-facing bound, and nothing here refuses a client. Ownership of a detached batch that does enqueue moves through three hands in a fixed order, and the first hand is not the worker. append_off_lock takes the ceiling and size partitions out of the buffers — dropping their bytes from the accounting — and returns them, all inside one sink-lock section and before emit_concurrent could register anything; the worker runs after pool.submit released ingest_bound, so a barrier can capture in the window where a pre-cut partition is out of the buffers and not yet in the in-flight set. Taking the exclusion in the worker to close that window would deadlock against quiesce_encodes, which holds it while waiting for that same worker. So the publish guard follows the route the BatchGuard already takes: it is created in submit, under the exclusion, with the same epoch, and travels with the queued batch; the worker’s detach consumes that pre-made guard instead of creating one, and a batch whose records detach nothing releases it unused at the end of the batch. No detach therefore happens outside a guard’s lifetime: the guard exists before the turn releases the exclusion, so every partition the batch detaches is in the in-flight set from the instant it leaves the buffers, and a cut captured at any point either drains it or waits for it. One guard per batch covers however many partitions that batch detaches — and a batch detaches them one at a time, each going to a different queue slot, so the guard cannot be moved into any single item. It is therefore a shared completion: an Arc created at submit holding the publish guard and a count, incremented once per detach and decremented once per completion (durable, requeued, parked, or dropped), settling the guard when the count returns to zero and the batch’s encode phase has ended — so a batch that detaches nothing settles at the end of its own loop, and one that detaches five settles when the last of the five finishes, whichever order they finish in. Each Detached item carries a handle to that completion, which is also why emit_concurrent needs no registration of its own; the queue carries the handle with the records. What the queue carries is not a Drained: that value is the audit buffer’s own snapshot taken under the miner lock beside the records, and a worker holds neither — it sees MinedRecords after their template events were already emitted into the audit sink. So the queue carries detached partitions only: its item is a Detached partition, which holds the detached records with their audit watermark and a handle to the batch’s shared completion. A Drained never enters it. The age sweep and the barrier keep writing theirs through write_ordered, synchronously on their own tasks, unchanged. The barrier must: run_cut needs its flush’s outcome (cut_ok) before it decides whether to stamp. The sweep stays there so that a write_ordered unwind keeps happening on the sweep’s own task, where #795’s stop-on-panic (§3.2) stops it and its JoinError counts cadence_panic; on the publisher thread the sweep would tick on under the latch, and whether the sweep survives an unwind is RFC 0054’s (publish unwind safety) decision to make, not this one’s. An earlier revision gave the item a Drained arm as well; slice D shipped without it (#867) because no producer fitted §3.2, and #875 settled the question by dropping it. The item carries the audit-sink position observed under the miner lock at detach time — the count of events the sink has accepted for that tenant when the partition left the buffers, which is at or above every event the partition’s records produced, since emission precedes the append. Before writing a detached partition the publisher requires the events up to that watermark to be durable, not merely gone from the buffer: a concurrent age drain can be holding exactly those events in an unfinished store write while this publisher sees an empty buffer, and publishing then would put records in Parquet ahead of their template events — the failure issue #302 fixed. So the audit sink tracks audit_durable_through, advanced only after a store write returns success — and advanced only over a contiguous prefix, which is the part a scalar alone cannot give: audit writes run unlocked and overlap, so a later write completing first must not carry the mark past an earlier one still in flight. The sink therefore keeps the set of in-flight ranges beside the mark and advances audit_durable_through to the end of the longest completed prefix with no in-flight gap below it; a write that completes above a gap records its range as done and moves nothing until the gap closes. A publisher whose audit_watermark is at or below audit_durable_through proceeds; one above it either writes the missing events itself, when it still holds them in the buffer (under the sink lock, ordered ahead of its records, the audit-first step write_ordered gives Drained), or waits on the sink’s condvar, which the mark’s advance notifies, re-testing its watermark on each wake until the mark covers it or the covering write fails. A failed in-flight write leaves audit_durable_through where it was and requeues its events, and every dependent record publish is refused and requeues with it — the same refusal write_ordered already makes when its audit half fails, now reaching publishes that did not take the events themselves. The publisher then settles the guard on the outcome: durable, or requeued into the buffers under the sink lock on a transient failure (where the next drain or flush covers it), or quarantined per record on a poison rejection, or, on an unwind, latched (§3.1’s guard rule) and settled. An audit write that fails transiently is the same refusal write_ordered already makes: the records are not published, they requeue, and the guard settles on the requeue.

The publisher thread needs a runtime handle, or it silently drops the graph feed. write_ordered hands every published batch to the RFC 0047 §3.3 flush-cadence graph emitter through tokio::runtime::Handle::try_current(), spawning the update off the publish path — and a bare std::thread has no current runtime, so try_current fails and the update is skipped with no error. Today that call always runs on a runtime worker; moving publishes to a dedicated thread would make every detached partition skip graph emission on exactly the deployments that configure OpenFGA. So the publisher carries a Handle cloned at construction and enters it (handle.enter()) for the duration of each write, which makes try_current succeed and leaves write_ordered untouched. The alternative — queueing graph updates back to the runtime through a receiver-owned channel — was rejected as a second queue with its own shutdown and overflow semantics for a feed that is already fire-and-forget; entering the handle is one line and keeps one code path for both callers. The verification plan covers it: the publisher test asserts the emitter is fed for a detached partition, which fails on a thread that never entered a runtime. The worker moves to its next record as soon as the enqueue returns, so it is never inside a PUT, the quiesce waits on encodes alone, and the PUTs settle outside the exclusion under quiesce_publishes like any other. At shutdown the order is the one the barrier already needs: stop the workers, quiesce the encodes, quiesce_publishes (which now also means the queue is empty), then join the publisher — a publish still queued at that point is never dropped, because the join waits for it. The publisher’s own unwind is defined, because a bare thread panic would strand every guard still in the queue and quiesce_publishes with them: the thread body runs each publish under catch_unwind, so a panic settles only the failing batch through its guard’s drop — latched (§3.1’s guard rule; its records stay in the WAL until a restart re-mines them, the stage-1 posture for every unwind) — and the same thread then drains every batch still queued back into the buffers as ready partitions, each park recording the current barrier_epoch (§3.1’s park rule) and releasing its guard as its records land, and only then exits; the coordinator respawns it on the next enqueue. quiesce_publishes therefore always terminates: every guard is settled by a durable write, a requeue, a park, or the failing batch’s own drop. The worker’s side of that window is defined too, because a send can fail after the publisher drained the queue and dropped the receiver, and the batch is then in no panic at all — a bare guard drop would settle it silently and the checkpoint would stamp over records in neither the buffers nor the store. So a failed send returns the item to the worker, which parks the batch into the buffers as a ready partition under the sink lock before releasing its guard — the same park the full queue takes, recording the current barrier_epoch with it — and then asks the coordinator to restart the publisher. Two rules make that safe. First, the slot is closed before the queue is drained: a publisher about to exit marks the slot closed under the publisher-slot mutex and only then drains its receiver, so a worker cannot succeed at an enqueue into a channel that is about to be abandoned — it observes the closed mark, parks, and joins the restart gate below. Second, the queue item’s destructor is the backstop: dropping an item that never settled requeues its records into the buffers under the sink lock and settles its share of the batch’s completion, so even an item lost to a drop no code path anticipated is data-safe rather than silently gone. The interleaving — enqueue racing the drain — is an acceptance leg, not an argument. Ownership is single at every instant: the worker parks first and only then retries, and a retry that succeeds removes the parked partition from the buffers under the same sink lock that put it there, so the records are in the buffers or in the queue and never in both. A retry that fails leaves the park standing and loops back through the gate. That ask is a claim, not a spawn, because several workers can observe the same closed channel: the coordinator holds the publisher slot — the channel sender and the thread handle — behind one mutex, and a worker that finds the channel closed takes the lock and re-reads the slot. The claimant is whichever worker finds the slot still holding the closed sender: it creates the new channel and thread, stores both, and releases. Every other worker finds a sender that is no longer the one it failed on, and simply retries its send against it — its batch is already parked, so a retry that fails again parks nothing twice and loops back through the same gate. The queue is owned by the slot throughout: the old receiver dies with the old publisher, which has already drained it into the buffers (§3.1’s unwind drain), so no item is stranded in a channel being replaced, and an item enqueued after the new sender is stored is the new publisher’s. No path releases a publish guard without the records being durable, requeued, or parked. RFC 0053’s requeue-on-unwind replaces only the failing batch’s arm of this; the thread, the queue, the drain-on-unwind and the guard-before-enqueue order stay. The barrier — reads the mark, drains both sinks into owned batches, registers that publish as in flight, and serialises each tenant’s miner snapshot state — the bytes write_snapshots would write — then releases. The flush of those batches, the snapshot file writes from those bytes, and the checkpoint all run outside the exclusion. Snapshot installation is serialised even so: every snapshot file carries that tenant’s own last folded frame from the cut (per-tenant marks, never the checkpoint’s global mark — a tenant with no frame near the cut must not be stamped past frames it has not folded). Those marks have a receiver-side source, since the WAL ledger says which frames exist and not which were folded — a frame whose group fsync failed is in the WAL, never acknowledged and never mined: each ingest turn advances the tenant’s folded horizon to its own frame offset after its mining step, the cut captures those horizons, and the WAL ledger stays the eligibility source only. The field that carries the mark is the artefact’s existing wal_high_water, and its meaning changes: today every tenant’s artefact holds the one global mark, after this RFC it holds that tenant’s own folded horizon. The snapshot format version is bumped (SNAPSHOT_VERSION 1 → 2) so the two readings can never meet in one reader: a version-1 artefact takes the existing unknown-version path — it is never a horizon; its tenant is discarded, logged by tenant and version byte, and fully replayed from the WAL — and that path is safe for one reason, which startup checks rather than assumes (§3.2’s RECLAIM record): a version-1 artefact can only predate this RFC, nothing was ever reclaimed before it (#793), so its tenant has no reclaim entry and pins at its oldest surviving frame like any tenant without a snapshot. Reading the old global mark as a folded horizon would in fact be sound — that mark was stamped only after every tenant’s frames at or below it had been drained and mined under the miner lock — but one byte with two meanings across two versions is a rule the next change breaks, and pre-production a persisted layout is broken rather than dual-read: the implementing PR carries the ! marker for the version bump, and no migration tooling is written. Each writer uses a unique temp name (<stem>.<mark>.snap.tmp, where <stem> is the percent-encoded tenant id snapshot_store already derives for <stem>.snap — the raw id is not a safe path segment — and <mark> is the byte offset; since today’s one fixed .snap.tmp could be truncated or interleaved by a concurrent cut before either rename — and the writer unlinks its own temp on any write, fsync or rename failure, with load_all_durable() removing any *.snap.tmp left by a previous process, so a failed barrier cannot leak one snapshot per attempt), the whole temp write, fsync and rename runs under a snapshot-install lock shared by the timer, the rotation hook and shutdown, and a writer installs only when its mark is not below the installed snapshot’s — so an older cut can neither overwrite a newer snapshot nor share its temp file, and the in-memory horizon advances only on an install that happened. Capturing the snapshot bytes at the cut is what makes S cut-consistent: it never runs past the mark, and every row at or below it is in the detached batches, so a turn admitted after the cut is neither in the snapshot nor suppressed by the max(X, S) gate on replay. The scope of that guarantee is stated, because it is narrower than “no duplicates”: a frame published after the cut and before the next checkpoint — by a size or age flush — sits above max(X, S) and is re-published on replay. That is RFC 0008 and RFC 0014’s existing at-least-once contract above the checkpoint, unchanged here and not claimed otherwise; a published high-water mark persisted with every flush would bound it and is recorded in §7 as the follow-on. Detaching only the file I/O keeps that property, and it means a slow or unavailable object store stalls ingest for the length of a drain and never for a PUT; holding the exclusion across flush_then_snapshot’s PUTs would let every retry during an outage recreate the local ingest outage, and would keep RFC 0053’s admission bound from running at all. The mark stays correct because every frame at or below it was drained into the detached batches, and every append after the release lands above it. The checkpoint is taken only after the detached publish and every publish registered before the cut have succeeded — quiesce_publishes returns the outcome of those registered publishes rather than merely waiting, and a failure in any of them means no stamp even though the cut’s own flush succeeded — by re-taking the journal mutex for the stamp alone; a failed publish requeues (RFC 0053 §3.2 makes that hold on unwind too) and no stamp happens. A failure is not sticky: the requeue consumes it by recording the barrier_epoch current when the records re-entered the buffers, and only a cut captured before that epoch — one whose drain could not have seen them — is refused by it. A cut captured after the requeue drained them itself and stamps normally; without that rule one transient failure would refuse every later cut for the life of the process.

Two cadences, so the barrier is not a force-all flush cadence. A cut drains every buffered partition, and running it on the 60-second housekeeping_secs would create a sub-target Parquet object per low-volume partition per tick — RFC 0014’s small-file hazard reintroduced by the reclamation path. So the barrier (cut, flush, snapshot, checkpoint) runs on its own barrier_secs, defaulting to the sink’s age trigger (300 s) and living beside housekeeping_secs in WalConfig, surfaced by wal_config() like the other WAL knobs, validated to at least one second, and owned by the receiver’s barrier task. For a partition holding data that old the age trigger would flush it anyway; a partition that received its first data just before the tick is written young, so the barrier does add at most one sub-target object per active partition per barrier_secs beyond RFC 0014’s policy — the explicit trade-off for a checkpoint that needs every buffered frame durable, and RFC0052.3 records the objects written per interval so it is measured rather than assumed. A busy partition still flushes on size between barriers. Housekeeping runs on housekeeping_secs against the last checkpoint, and an eligible closed segment is reclaimed by the first capped pass that reaches it — at most barrier_secs + housekeeping_secs after its publication when the backlog is within the cap, later ones waiting through further passes, and segments a pinned tenant or a failed snapshot holds retained by design. The current append segment is never unlinked, and rotation is checked only on an append today, so the barrier task also calls Journal::rotate(Discretionary) when the current segment’s age exceeds segment_age_secs and it holds a frame — an idle node’s last segment then rotates, and its bytes are reclaimed within barrier_secs + housekeeping_secs + segment_age_secs, which RFC0052.3’s idle leg asserts. That rotation is ordered with the cut, not beside it: Journal::rotate(Discretionary) returns nothing the cut could use, so the idle path rotates first, under the barrier exclusion, then captures the cut, and the cut’s mark is what every cut’s mark is — last_durable, the last acknowledged turn’s own frame offset — never the rotation boundary. The exclusion is what makes the order mean something: no turn can run between the rotate and the mark read, so every frame at or below the mark is in the closed segment and every frame appended after the release is in the new one, above it. A frame the WAL holds but never acknowledged — a turn whose group sync failed — is never a mark, and if an earlier such frame sits below a later acknowledged turn’s offset it is suppressed on replay like any frame below the gate, which loses nothing acknowledged: the client was told to retry it. RFC0052.14 asserts the rotate-before-cut order and the mark’s source.

Acquisition order is fixed, or the two locks deadlock. ingest_bound takes the miner lock today before the pool submission and the last_durable update; if it took the miner lock first and then waited for the exclusion while the timer held the exclusion and waited for the miner lock, neither would proceed. So every ingest turn acquires the exclusion before the miner lock, the timer follows the same order — exclusion, then miner lock, then the last_durable mutex — and the exclusion is never requested while the miner lock is held.

The age sweep participates in the same exclusion. It drains under the miner lock alone today, so a sweep could begin after the timer’s quiesce and before its stamp, leaving a drained-but-undurable batch outside the buffers that the barrier then reads as empty; quiesce_publishes waits only for a sweep already in flight and prevents no new drain. The sweep’s drain therefore takes the exclusion in shared mode — before the miner lock, which means the drain helper changes rather than the shared hold being added inside today’s with_miner(drain_aged): taken inside it, the sweep would hold the miner lock waiting for the shared exclusion while the timer held the exclusive lock waiting for the miner, and both would deadlock — held only across the drain and the in-flight registration, and released before the off-lock publish, whose PUTs can block on the store — and the timer takes it exclusively: no sweep drains inside the barrier, a drain in progress completes before it, and quiesce_publishes then waits for the publishes already registered. A slow or unavailable store stalls ingest for the length of a drain, never of a PUT, which is the WAL-local durability this RFC exists to preserve.

The mark is then read inside that turn and after the quiesce — last_durable() at that point, the offset the receiver’s acks are gated on — so every frame at or below it has finished its turn: mined under the miner lock the barrier now holds, drained, and its publish waited for. Frames above it may well exist — appended and synced by turns admitted after the cut, waiting on the exclusion — and that is the point: they have not touched the miner state the snapshot serialises, so the snapshot excludes them and replay re-mines them. Reading the mark before the quiesce, or outside the turn, reopens the window from the other side: a frame at or below the mark whose encode or publish is still in flight.

And the stored mark must not over-cover. The group-commit sync reports the WAL’s EOF, and CommitCoordinator::flush captures covered_seq before it locks the journal, so waiter A’s outcome can carry an offset that includes frame B, appended later by a turn that has not yet run — while B is made durable by that same outcome and will be acknowledged once its own turn completes. If A stored that EOF as last_durable, a barrier between A’s turn and B’s would checkpoint across B, and recovery would suppress a frame nothing had mined. So last_durable advances to the turn’s own frame offset — the WalOffset its append returned, carried through CommitOutcome beside the durable EOF — and never to the EOF itself. Turns are sequential, so the stored mark is a contiguous-success high-water by construction, and that is the only mark the barrier reads.

Contiguity is over acknowledged frames, which is the property §3.4 needs. A turn that fails still releases the gate — IngestGateGuard drops on every exit — so its frame can sit below a later successful turn’s mark, and a checkpoint at that mark covers it. That is safe: a failed turn’s batch was never acknowledged (the handler answers 5xx and the client re-sends), so suppressing its frame on replay loses nothing acknowledged, and the re-sent copy is a fresh frame above the mark. Tracking the highest contiguous successful sequence instead would hold the checkpoint back for a frame nobody was promised.

The cost is explicit: ingest stalls for the cut’s duration — a quiesce and two drains, no I/O — once per barrier_secs, comparable to the stall rotation already imposes, now on a timer. barrier_secs is therefore the knob trading reclamation latency against that stall; housekeeping_secs schedules only the capped unlink pass, whose own stall §3.7’s per-pass cap bounds.

See §3.7 for the ownership path and the exact signatures, which the barrier does not have today:

if barrier_succeeded and high_water is Some(mark):
    journal.checkpoint(mark)            // §6.7, monotonic; Result
      on Err  -> log, do NOT advance; nothing past the PREVIOUS mark is reclaimed

Three properties come free from existing code and must not be reimplemented:

• Monotonicity and idempotence — checkpoint rejects a regression and no-ops on a re-assert, so a repeated stamp at the same mark is safe.
• Fail-closed — on a sidecar write error the in-memory checkpoint is not advanced, so nothing past the previous mark is reclaimed — segments already eligible under that mark still are (RFC0052.1) — rather than risk a post-crash duplicate.
• Skip-on-retain — when either sink retains anything, flush_then_snapshot returns false and no checkpoint is attempted. The WAL keeps the frames and the next start re-mines them.

A cadence panic latches the barrier until RFC 0053. #795 stops the age sweep after a panic, which was a safe interim only while nothing else could stamp: this RFC’s timer can, and after a panic inside write_ordered the drained batches are gone and the publish guard has dropped, so a timer barrier would see empty buffers and no in-flight publish and stamp past records that never reached Parquet. So a panic on any path that holds records outside both the buffers and Parquet sets a cadence_failed latch — set on the unwind path itself, by a guard’s Drop that runs before the publish guard is released, never by the joining task observing the panic later, or a barrier could acquire the exclusion, see empty buffers and stamp in that window. Three paths hold records that way, and all three set it: the age sweep’s step, from the in-flight registration drain_aged takes through write_ordered — one guard wraps the whole step, so a panic mid-drain, with records in a half-built batch, latches too; the barrier’s own detached flush; and an encode worker. The worker is the one the earlier drafts missed: emit_concurrent runs after ingest_bound has acknowledged the frame, and the pool’s BatchGuard today only settles the pending count on unwind, so a worker panic mid-batch leaves quiesce seeing no pending work while the batch’s unemitted remainder is in neither the buffers nor Parquet — a barrier after it would stamp across the frame. The BatchGuard therefore sets the latch when it drops unwinding, before it decrements, so worker unwind feeds the same failed-cut path as a cadence panic in this stage; RFC0052.10 is not gated on RFC 0053 for it. The signal has to be concrete, because nothing else can carry it: the pool’s workers are detached std threads whose join handles are kept only for drop, so a worker panic is invisible to the receiver while it serves. The latch is two shared atomics owned by the pipeline (SharedPipeline holds the Arc) and handed at construction to the encode pool, to the sink’s publish guards and to the barrier task: barrier_epoch: AtomicU64, the number of the next cut, incremented at every cut capture under the exclusion; and one packed AtomicU64 carrying the failure state — the failed epoch in the high 32 bits, a failure generation in the low 32 — rather than two words. Its clear encoding is u32::MAX in the high half with the generation zero, and that matters: a default-zero word would read as “epoch 0 failed”, which is at or below every cut’s epoch and would refuse every cut on a node that had never panicked. Epoch u32::MAX is reserved and never assigned to a cut — barrier_epoch stops before it, as the overflow note below says — so the sentinel cannot collide with a real failure, and a clear restores exactly that word (sentinel high, generation zero), not merely a high half. Two words would not be safe to clear: a report landing between a reader’s two loads can be missed, and a clear written against the epoch alone would then erase a failure that was never acted on. Packed, the operations are exact. A guard reports with a CAS loop that lowers the epoch to its own and bumps the generation in the same word, so no report is lost to a concurrent one. A cut captures the state with a single acquire load and decides against that word. A clear is a CAS on the exact word the cut captured, so a report that landed in between makes the CAS fail and the latch stays set. Stage 1 never clears — only a restart does, as §3.1 says below — so the clear arm exists for RFC 0053 §3.2, which defines when a cut may perform it; what stage 1 needs from the packing is that its capture and its two checks read one indivisible value. The generation wraps at u32::MAX and that is harmless: it is compared only for equality inside a CAS, never ordered, so a wrap can at worst make one clear spuriously fail, which leaves the latch set — the safe direction. The epoch half bounds a node at 2^32 cuts, about 40,000 years at the 300-second default, and a node that somehow reached it would stop stamping rather than wrap into a stale comparison. A guard reads barrier_epoch when it is registered — submit for a BatchGuard, begin_publish for a publish guard, both under the exclusion, so the read is ordered against the capture: a guard registered before cut E’s capture carries E, one registered after it carries E + 1. For the pool that means the BatchGuard is constructed in submit, with its epoch, and travels with the queued batch — today the worker constructs it after dequeue, and a batch queued before cut E’s capture but dequeued after it would then read E + 1 and a later panic in it would fail the wrong cut. submit already increments the pending count under the exclusion; the guard and epoch are assigned there too, and the worker only drops what it received. Its Drop tests std::thread::panicking() and, if so, lowers failed_epoch to its own epoch (a CAS keeping the minimum) with Release ordering before it decrements its count — the pending count for BatchGuard, the in-flight count for the publish guard — and it is that decrement, under the count’s mutex and condvar, that wakes quiesce or quiesce_publishes. flush_then_snapshot loads failed_epoch with Acquire at both of its checks, and a cut with epoch E fails when failed_epoch ≤ E. The epoch is what keeps the latch honest about scope: a guard carries the epoch of the first cut whose mark could cover its records, so a panic in a publish registered after cut E (its frames are above E’s mark, §3.1’s invariant) fails no cut at or below E — the cut in flight proceeds — and fails every cut from E + 1 on. The race contract follows from the order: a barrier that observes a count at zero after a panic observed a decrement that happened after the store, so it observes the epoch; there is no schedule in which the count settles and the latch is still clear. RFC0052.1 holds that race directly, with the panic injected in a worker and the barrier started concurrently. In this stage the dropped batch’s records exist only in the WAL, so nothing can clear the latch short of re-mining them: every barrier with an epoch at or above failed_epoch neither checkpoints nor snapshots — the frames stay in the WAL and a restart replays them — the state is exported (RFC0052.7) and only a restart clears it. That is the conservative behaviour, stated as such; RFC 0053’s requeue-on-unwind is what makes a later cut able to clear it, by draining the requeued batch, and it defines that clear. The check lives inside flush_then_snapshot, the one function the timer, the rotation hook and shutdown all call, so every stamping caller honours it rather than the timer alone, and it is made twice: before the cut, and again immediately before the snapshot install and the checkpoint, after quiesce_publishes has returned. The second check is not redundant. A publish registered before the barrier began can panic while the barrier is waiting on it, and the latch it sets lands after the first check; a latch observed at the recheck is a failed cut — no install, no stamp; the cut’s own batches are in the store already and the WAL still holds every frame, so a restart replays them into the max(X, S) gate like any other. The same unwind guard also records that publish’s outcome as failed, so quiesce_publishes().all_ok() is false for it independently of the latch: the recheck defends the ordering, the outcome defends the data, and either alone refuses the stamp. RFC 0053’s requeue-on-unwind is what removes the latch, by making every one of those panics lose nothing to stamp past.

A snapshot write failure is deliberately not a checkpoint blocker: flush_then_snapshot logs it and still returns true, because the data is in the store and the snapshot only governs replay depth. The truncation floor below is what keeps that safe for the WAL. It is safe for Parquet only if recovery honours the checkpoint on the Parquet side: after a restart, frames at or below X may be re-fed to the miner when the snapshot lags (S < X), but must never be re-published to the record sink, or every row between S and X lands in Parquet twice. recovery.rs describes exactly that split and today gates only the miner on S; the Parquet-side gate on X is a requirement of this RFC, not an assumption, and RFC0052.10 asserts it — a restart after a failed snapshot write and an advanced checkpoint produces no duplicate rows.

3.2 Housekeeping on the timer that already exists

WalConfig::housekeeping_secs (default 60) is set in every test and read by nothing. RFC 0008 §6.7 says “the timer lives in the caller”. The caller is the receiver role, on its own interval:

every barrier_secs (default: the sink age trigger, 300 s):
    if failed_epoch <= barrier_epoch: skip // §3.1's latch; housekeeping below still runs
                                           // (RFC 0053 §3.2 replaces this guard with
                                           //  proceed-and-decide once its requeue exists)
    cut = with_barrier_exclusion:          // a NEW pipeline lock, not the miner
                                           // lock and not the gate — see §3.1
        quiesce_encodes()                  // the barrier's prologue: every pre-cut
                                           // batch encode-complete (§3.1)
        mark = last_durable()              // read AFTER the quiesce, INSIDE the turn
        batches = drain both sinks         // owned; registered as in flight
        snaps = serialise miner snapshots  // cut-consistent S (§3.1)
        epoch = barrier_epoch.fetch_add(1) // this cut's epoch (§3.1)
        (mark, batches, snaps, epoch)      // release the exclusion here
    run_cut(cut)

on a rotation (inside the ingest turn that observed the segment change):
    cut = capture as above, mark = prev    // the turn already holds the exclusion;
                                           // no store I/O in the turn (§3.1)
    hand cut to the barrier task           // one pending slot: fills it, or coalesces
                                           // into the pending cut (§3.1); never blocks

run_cut(cut):
    cut_ok   = flush(cut.batches)          // store I/O OUTSIDE the exclusion
    prior_ok = quiesce_publishes().all_ok(cut.epoch)
                                           // publishes registered BEFORE the cut: outcome,
                                           // not a wait; always evaluated, so a failed cut
                                           // flush cannot skip their outcome and requeue
                                           // path. False iff one of them failed and requeued
                                           // at an epoch ABOVE cut.epoch — a requeue this
                                           // cut's drain could not have seen (§3.1); a
                                           // publish registered after the cut holds only
                                           // frames above cut.mark, so it is neither waited
                                           // on nor stamped past
    ok = cut_ok and prior_ok and failed_epoch > cut.epoch
                                           // the recheck (§3.1): a panic in a guard of
                                           // this cut's epoch or lower is a failed cut;
                                           // a later epoch's panic fails later cuts
    if not ok and pending slot is filled:  // strict ordering (§3.1): the pending cut
        invalidate pending                 // folded this cut's frames; discard its snaps,
                                           // merge its batches into the buffers, withdraw
                                           // its mark, re-capture under the exclusion
    if ok and cut.mark is Some(_):
        install_snapshots(cut.snaps)       // under the install lock, monotonic in mark;
                                           // failure logged, not a blocker (§3.1); with
                                           // no mark the previous snapshots stay
    if ok and cut.mark is Some(m):
        coordinator.checkpoint(m)          // re-takes the journal mutex for the stamp alone
      on Err -> log, do NOT advance (fail-closed, §3.1)

every housekeeping_secs (default 60 s), in its own task:
    coordinator.maintain(snapshot_horizons(), config.max_unlinks_per_pass):
        plan    = journal.lock().housekeeping_prepare(horizons, cap)
                                           // ledger half under the guard: horizons applied
                                           // (capped), floor derived, partials then segments
                                           // popped and marked reclaiming (§3.2, §3.7)
        outcome = file half on plan        // guard RELEASED: RECLAIM write + fsync + rename +
                                           // parent fsync, then unlinks, then parent fsync
        journal.lock().housekeeping_commit(outcome)
                                           // ledger half under the guard: removed entries
                                           // leave the ledger, failed ones return/stay (§3.2)
      on Err -> log; the next pass retries (nothing was unlinked past the bound)

Housekeeping does nothing until the witness exists. The two timers are independent, so on a legacy root the housekeeping timer can fire before the first barrier has upgraded the sidecar — and reclaiming there would unlink segments under a version-1 checkpoint, the one shape §3.2’s matrix reads as “nothing was ever reclaimed”. So a pass whose root still has a version-1 CHECKPOINT, or no RECLAIM record, plans no segment and writes no record: it returns a progress with zero segment counts, counted as a skipped pass with its reason on the existing cadence counter (error.type carries the reason, §3.5’s rule). The gate is on segment planning and the record write, and on nothing else: the partial sweep still runs on every pass, witness or not. A .wal.partial is debris from a rotation that never installed a segment — no reader has ever been able to depend on it, no reclaimed_through accounts for it, and removing it loses nothing — so gating it would let rotation debris created before the first checkpoint survive forever, against RFC0052.4’s one-pass clearance and RFC0052.16. Partials are popped first within the cap as always, and the cap’s remainder simply goes unused while the gate holds. The first barrier’s checkpoint establishes the witness and the next pass reclaims segments normally. This is not a new gate so much as a narrower one: today’s housekeeping is already a no-op before the first checkpoint.

The timers have the age sweep’s shutdown lifecycle but not its task. Both loops are owned by the receiver, signalled by the same shutdown watch the sweep observes, and joined before the final flush and snapshot, in the order ReceiverHandle::shutdown already joins the sweep; they hold no pipeline or journal handle after the join. They are separate tasks from the sweep, and from each other: the sweep’s stop-on-panic (#795) must not stop reclamation, and a panic in a barrier step — or in the sweep’s own step, or in an encode worker (§3.1’s three latching paths) — sets cadence_failed and leaves the housekeeping loop running housekeeping-only passes, which is the branch the pseudocode’s skip relies on. That promise needs an owner, since the per-batch guards cover only what a batch holds and a panic between them would still kill the task: each task runs every tick under catch_unwind. A barrier tick that panics stores failed_epoch = min(failed_epoch, this tick's epoch) — the same CAS the guards use, so the cut and every later one are refused until §3.1’s clear — invalidates the pending slot by the rule above, and the task takes the next tick. A housekeeping tick that panics counts on the existing cadence counter with error.type = cadence_panic (§3.5’s naming rule: the class rides error.type, not a new metric) and takes the next tick; any uncommitted plan is re-planned by the next housekeeping_prepare, which is exactly the case §3.7 already defines. At shutdown a JoinError from either task is logged and read as a failed cut — no stamp, nothing assumed drained — rather than as a clean join. The sweep keeps #795’s stop-on-panic until RFC 0054 (publish unwind safety) makes it survivable, and keeps its writes on its own task for that reason (§3.1: the publisher queue carries detached partitions only); what changes here is that stopping the sweep no longer stops reclamation or the barrier. The sweep’s panic is the one #795 already stops the sweep on; what this RFC adds is that the same unwind now latches before the sweep’s publish guard drops, so the barrier that #795 could not foresee cannot stamp past the batch the panic dropped, whether the barrier was waiting on that publish or arrives later. A timer that was not joined at shutdown could race the shutdown reclamation or keep the WAL alive past the handle, which is why the ordering is part of the design rather than of the implementation.

retain_floor is the MINIMUM over per-tenant snapshot horizons, not the latest one. write_snapshots persists one snapshot per tenant and can partially succeed, so there is no single “latest” mark: taking the highest would let housekeeping unlink frames above a lagging tenant’s horizon, and that tenant’s miner state could then not be rebuilt. CLAUDE.md §3.7 is explicit that every path touching data is tenant-scoped, and a per-node floor is exactly the bolted-on form it forbids.

The minimum is already this codebase’s stated contract, not a new invention — recovery.rs’s stale-gap detector names it:

the §6.7 retain floor (min over tenant horizons) keeps any segment holding frames above the floor, and a lagging tenant’s own S.segment is protected by its own membership in the min

Under tenant-aware retention that argument survives only if each tenant’s snapshot horizon S is that tenant’s own last folded frame offset, not the cut’s global mark — a global mark’s segment can hold none of the tenant’s frames and be legitimately unlinked, and the detector would then report external mutation for S < X. So snapshots record per-tenant horizons, and the detector is restated on the RECLAIM record rather than on segment presence, because under per-tenant horizons S < X is the normal case and S’s own segment is legitimately unlinked once the tenant’s horizon covers it (§3.2’s inclusive rule). Today’s predicate — S < X and S.segment unseen — would fire on every idle tenant after its last segment is reclaimed. The amended detector asks the record instead: S.segment absent is explained when the tenant’s recorded reclaimed-through is at or above S, and it is S > reclaimed-through with the segment absent that is external mutation. The record’s own comparison (a restorable horizon below its reclaimed-through halts) is the other half of the same check, so the two read one witness.

That detector’s no-false-positive property depends on the minimum, so the earlier draft’s “latest” wording would have broken stale-gap detection as well as losing data.

A tenant with WAL data and no valid snapshot pins the floor at its oldest surviving frame. It does not halt reclamation outright — an earlier draft said “reclaim nothing”, and with shared segments and tenant churn that is a permanent halt: a tenant that wrote once and whose snapshot never landed would block every other tenant forever. What hazard #5 actually requires is that that tenant’s frames survive, since with no snapshot a restart must re-mine all of them. So eligibility is per segment and tenant-aware, not a single global bound — a global “highest offset below the minimum” rule would let one unsnapshotted frame pin every later segment, even those holding only other tenants, which is the permanent growth this section exists to end. The rule, stated once: a closed segment is reclaimable when its highest offset is at or below the checkpoint (inclusive, a post-append horizon), and for every tenant with a frame in it that tenant’s last offset in that segment is at or below its valid horizon — inclusive at the horizon, because the horizon frame is the last frame the snapshot folded and the snapshot recording it is durable: SnapshotHorizons is built from the SnapshotLedger, which holds only marks whose install returned Ok after the parent fsync, so every horizon housekeeping_prepare ever sees is a durably installed one, and “inclusive once the install is durable, strict while it is not” collapses to inclusive. An earlier draft compared strictly so that S‘s segment stayed present for the stale-gap detector; that left the segment holding an idle tenant’s horizon frame permanently ineligible — the idle node RFC0052.3 requires to reclaim its last segment could not — and the detector no longer needs the segment, because the RECLAIM record (below) says why it is gone. A pinned tenant has no horizon, so exactly its own segments are retained — the pin is strict: a pinning tenant’s last offset in a segment at or above its pin is a frame the tenant needs, and equality can never unlink it — while segments holding only other tenants’ covered frames are reclaimed whatever their offset. RetainFloor is the reported summary of that rule (the minimum over horizons and pins), not the predicate. The pin lifts the moment a valid snapshot for the tenant is written, since its horizon then replaces the pin. So the three comparisons are: checkpoint inclusive, horizon inclusive, pin strict. RetainFloor::Min carries no comparison of its own — it is the reported minimum, and today’s min(checkpoint, floor) bound in Wal::housekeeping, which compared a segment’s highest offset inclusively against that minimum, is replaced by the per-segment rule rather than kept beside it. None is passed only where no snapshot consumer exists at all.

The ledger is WAL-owned, so the floor is derived where the frames are. Neither the Journal API nor RecoveryReport could otherwise tell the receiver which tenants have surviving frames or at what offset. The WAL learns a frame’s tenant from the TenantOtlpBatch prefix it already frames in append_batch, and rebuild_ledger() rebuilds the ledger from every surviving frame’s prefix at the end of recovery; the receiver passes only what it knows — the per-tenant snapshot horizons, or “no consumer” — and the WAL derives RetainFloor from the two and reports it. The accounting is frame-kind-specific: only TenantOtlpBatch frames carry tenant membership and drive the per-tenant horizon rule, because they are what a restart re-mines. AuditEvent frames are governed by the checkpoint alone — their consumer is the audit sink, which the barrier drains before it stamps, so an audit frame at or below X is published and reclaimable and one above X is retained by the checkpoint rule with no snapshot horizon involved; a segment holding only a pinned tenant’s audit frames is reclaimable once below X, since the pin protects re-mining input and audit frames are not that. Membership is per surviving segment, so the ledger has a lifecycle. Each segment carries, per tenant, the WalOffset of that tenant’s first and last frame in it — the first is what Pinned.offset needs, the last is what the eligibility rule compares the tenant’s horizon with, since a horizon between the two must not unlink the later frames; a membership set alone could recover neither, and a segment-wide boundary would be a different contract — rebuilt at open from the recovery walk, the last updated on every live append; a tenant’s oldest surviving frame is its first offset in its oldest surviving segment. A tenant leaves the ledger when its last surviving segment is unlinked, which can only happen once its snapshot horizon has passed those frames; nothing else removes it, and nothing needs to. recover returns entries only for tenants that have a .snap, so without this ledger an empty or partial snapshot set would be indistinguishable from “no consumer” and read as None.

And the floor may only be derived from snapshots known to be durable. snapshot_store::write renames the new snapshot into place before its parent-directory fsync, so a failure there leaves the file visible to this process while the write returned an error. A floor derived by listing .snap files would then trust a horizon that may not survive a crash, and housekeeping would unlink segments on the strength of it — losing exactly the frames the floor exists to retain.

So the derivation uses the in-memory record of snapshots whose write returned Ok, not the directory contents. That record has an owner: a receiver-side SnapshotLedger, restored from load_all_durable() at startup, updated by write_snapshots’ per-tenant report — returned instead of short-circuiting on the first error into a single Result<()> — so it advances exactly the tenants whose new snapshot completed and keeps the previous durable horizon for the rest, and read as the SnapshotHorizons handed to maintain. No path re-reads the files after a failed fsync. A tenant whose snapshot write failed keeps its previous durable horizon if it has one and otherwise pins the floor at its oldest surviving frame, per the rule above — conservative, and self-correcting on the next successful write. Reading the directory is legitimate only at startup, before anything has been reclaimed in this process’s lifetime — and even then only after one more step.

The startup listing is revalidated by a directory fsync before it is trusted. The in-process rule covers failures this process observed, but a previous process may have renamed a snapshot into place, failed the parent-directory fsync, and exited cleanly; the entry is then still visible to a restarted process, which would load it as a horizon, reclaim on the strength of it, and lose the frames when a later machine crash drops the never-durable entry. So before any listed .snap is used as a horizon the snapshots root and its parent are fsynced once — snapshot_store::write creates the directory with create_dir_all and fsyncs only the child, so the directory’s own entry in wal_root is not durable either until this RFC adds a parent fsync on creation — which makes every entry the listing saw durable, the one property the listing lacks. This is an explicit operation, snapshot_store::load_all_durable(), which fsyncs before it lists, treats a missing snapshots_root as the empty store (a cold start has no directory yet, and that is not a durability failure), and fails startup when either fsync fails, as a RecoveryDriverError — so an implementation cannot keep trusting the bare listing. An earlier draft discarded every snapshot and started empty instead; that is not fail-closed once reclamation is live, because a tenant’s older durable snapshot may be the only thing that can rebuild its state, the frames below that horizon being gone, and a node that starts empty then replays a tail that cannot restore it. A disk that cannot fsync the snapshots directory at startup is a fault to surface, not to continue past. One directory fsync rather than a manifest, because a manifest would need the same fsync to be trustworthy itself. Durability is necessary and not sufficient: snapshot_store::load_all returns raw bytes, and a horizon is admitted only from a snapshot that decodes and restores. A listed file that fails either is never a horizon — but whether the node may then continue needs durable evidence, because once reclamation has run, the frames below that tenant’s previous horizon are gone and a pin at its oldest surviving frame cannot rebuild the state they held. So housekeeping persists, per tenant, the horizon it reclaimed under — a RECLAIM sidecar beside CHECKPOINT, under the same contract as that file and not a looser one, because startup’s fallback is exactly as trustworthy as this record:

• A fixed format, in the house style. CHECKPOINT is a 32-byte record — 4 B magic b"OWCK", 2 B little-endian version, 2 B reserved flags, then its payload — and RECLAIM follows it rather than inventing a shape. The file is two fixed-size slots behind one 32-byte file header: 4 B magic b"OWRC", 2 B version (= 1), 2 B reserved flags, 8 B slot length, then the two capacities the geometry was computed from — u32 max_tenants, u32 max_unlinks_per_pass — then u32 CRC32-C over bytes [0..24) of the file, the header’s own preceding bytes, and 4 B reserved, 32 B in all. The capacities are in the file because the configuration can change across restarts while the file cannot grow: Wal::open compares the configured pair against the stored one and rebuilds the file at the larger geometry when either configured value exceeds what it was built for, rather than refusing: refusing would strand a root whose operator legitimately raised a limit the other sidecars can grow into. The rebuild is crash-safe and is the one place this record does use a temp: the new geometry is written whole to RECLAIM.new — both slots, the live one’s contents copied into the wider dictionary stride at the same index, so every slot id survives unchanged, and the other zeroed — fsynced, renamed over RECLAIM, and the parent fsynced, before anything else reads it. A crash at any point leaves either the old file or the new one, both complete, since the rename is atomic and the temp is never read; a RECLAIM.new left by a crash is truncated by the next rebuild and swept by §3.2’s selector, which already reserves that namespace. Growth is therefore an open-time cost paid once, not a per-pass allocation, and the steady-state commit still never allocates. A configured value at or below the stored capacity opens normally and simply uses less of each slot; the file is never shrunk, since a smaller geometry buys nothing and a rewrite risks something. Each slot is [u64 generation][u32 entry_count][u32 planned_count][u16 header_flags][u16 consumer_mode][u32 next_slot_id] (24 B), then three fixed arrays — the dictionary, the entries and the planned records — then a u32 CRC32-C and 4 B reserved (8 B). The checksum covers bytes [0 .. slot_len - 8) of that slot — everything before the trailer, never the trailer itself, since a checksum cannot cover its own bytes.

  Nothing in a slot is walked by a count: a position is an identity. The entry array holds max_tenants entries at a fixed 32 B stride, and entry i belongs to dictionary record i — the same id, by position — so an entry carries no tenant field at all and occupancy is the entry’s own occupied flag. That is what makes an id survive a tombstone: a removed tenant leaves its entry zeroed in place, the ids above it do not move, and nothing renumbers until a rebuild. A dense [0, entry_count) prefix cannot do both — a hole either skips an id or shifts every id above it — which is why entry_count is a count of live entries for reporting, never a length to walk.

  The planned array is the same shape for the same reason, and costs nothing to make so: max_unlinks_per_pass records at a fixed 24 + 32 × max_tenants stride, each with a fixed array of max_tenants pairs where pair i is dictionary record i’s, so a pair carries no tenant field either and its presence is its own flag. The worst case was already preallocated — a variable stride would occupy the identical bytes while making a record findable only by walking its predecessors — so planned_count and a record’s tenant_count are likewise reported counts, not lengths. A reader that trusted either as a length would be reading a torn or stale slot’s arithmetic; it reads the fixed strides instead, and the slot’s CRC is what says the bytes are the ones the writer wrote. header_flags carries four witness bits and nothing else: checkpoint_armed (bit 0), checkpoint_seen (bit 1), published_seeded_armed (bit 2) and published_seeded_confirmed (bit 3), with bits 4–15 reserved and zero. The second pair is RFC 0053’s, amending this header: its PUBLISHED seeding needs a durable two-state witness and this is where the other witnesses live, so it takes two bits of the reserved range rather than a field of its own — no width change. Its meaning is the one the checkpoint pair already establishes: armed is written durably on the accepting start before any batch is admitted, confirmed at the next record write after the first PUBLISHED write succeeded, and once confirmed it never clears. The crash rows follow the checkpoint witness’s exactly — armed with PUBLISHED absent is the window before that write, and the next start seeds again correctly; armed with PUBLISHED present is the ordinary post-write state, promoted to confirmed at open — while RFC 0053 §3.2 owns what the seeding itself does. consumer_mode is 0 unrecorded, 1 Known, 2 NoConsumer (§3.2), so “unrecorded” is a value rather than an absence.

  Tenants are referenced through a per-slot dictionary, not inline. TenantId is not a UUID: ourios-core defines it as a validated string of 1 to MAX_TENANT_BYTES = 128 bytes of ASCII graphic characters, so a 16-byte key would either truncate it or collide two distinct tenants onto one entry — and this record is read as proof of loss, where a collision silently pins the wrong tenant. Each slot therefore opens with a dictionary of max_tenants fixed records, [u16 len][128 B key bytes][2 B reserved] (132 B), len naming how many of the key bytes are significant and the rest zero; an unused dictionary record has len = 0. The key field is 128 bytes, and the governing spec is RFC 0048 §3.1, whose title says what it does — “Tenant id grammar (amends RFC 0046 §3.1)” — setting a tenant id at 1 to 128 bytes of ASCII graphic characters with :, # and / excluded. RFC 0046 §3.1’s 256 is the amended text, not the live bound; ourios-core’s MAX_TENANT_BYTES = 128 is the spec, not a stricter local choice, so sizing the record to 256 would double a preallocated file for a length no boundary admits.

  The frame codec is amended to match. ourios-wal’s TenantBatch::MAX_TENANT_BYTES is 256 today, its comment still citing RFC 0046 §3.1, so a frame can carry a 129-to-256-byte tenant that no request boundary would have produced and that a 132 B dictionary record cannot represent. This RFC lowers it to 128 on encode and decode, and a replay that yields a longer tenant fails closed at open, naming the frame offset and the length rather than truncating a key or silently dropping a frame. Pre-production, that is the whole migration — no tooling, no dual-read — and RFC 0053 states it the same way. Every other reference is a u16 slot id into it. The mapping is injective by construction — each id names one record, each record holds the exact bytes and length of one tenant id — so it round-trips to the original TenantId with no normalisation and no lossy comparison.

  An entry is then [u16 flags][u16 mode][4 B reserved][24 B WalOffset] (32 B), its position naming its tenant and bit 0 of its flags marking it occupied — the mode per entry as well as per root, since §3.2 reads entries by the mode they were written under; a WalOffset is its 16 B segment UUID plus its 8 B byte offset, 24 B, wherever it appears. A planned record is [16 B segment UUID][u16 tenant_count][4 B reserved] [u8 flags][1 B reserved] (24 B) followed by its fixed array of max_tenants pairs, each [u16 flags][6 B reserved][24 B WalOffset] (32 B). The record’s flags are at byte 22: bit 0 uncertain, bit 1 occupied — a planned array position that no segment occupies is zeroed, which is how a pass shorter than the cap is represented. A pair’s flags bit 0 is present, and a pair’s position is its tenant, so an absent pair is simply a zeroed 32 B run.

  Every field is at a fixed offset, little-endian, and every reserved byte is zero and checked. Two implementations must produce the same bytes, so the layout is pinned rather than described. All multi-byte integers are little-endian, matching the OWCK record and the segment header, which encode their u16s with to_le_bytes today. Offsets, relative to the start of each structure:

  Structure	Offset	Size	Field
  File header	0	4	magic b"OWRC"
  	4	2	u16 version (= 1)
  	6	2	reserved, zero
  	8	8	u64 slot_len
  	16	4	u32 max_tenants
  	20	4	u32 max_unlinks_per_pass
  	24	4	u32 CRC32-C over bytes [0..24)
  	28	4	reserved, zero
  Slot header	0	8	u64 generation
  	8	4	u32 entry_count
  	12	4	u32 planned_count
  	16	2	u16 header_flags (bit 0 checkpoint_armed, bit 1 checkpoint_seen, bit 2 published_seeded_armed, bit 3 published_seeded_confirmed, bits 4–15 zero)
  	18	2	u16 consumer_mode (0 unrecorded, 1 Known, 2 NoConsumer)
  	20	4	u32 next_slot_id — the id high-water mark
  Dictionary record	0	2	u16 len
  	2	128	key bytes, len significant, remainder zero
  	130	2	u16 flags: bit 0 tombstoned, bits 1–15 zero
  Entry (array of max_tenants, entry i = dictionary record i)	0	2	u16 flags: bit 0 occupied, bits 1–15 zero
  	2	2	u16 mode
  	4	4	reserved, zero
  	8	24	WalOffset: 16 B segment UUID then u64 byte
  Planned record (array of max_unlinks_per_pass, stride 24 + 32 × max_tenants)	0	16	segment UUID
  	16	2	u16 tenant_count (reported, not a length)
  	18	4	reserved, zero
  	22	1	flags: bit 0 uncertain, bit 1 occupied, bits 2–7 zero
  	23	1	reserved, zero
  Planned pair (array of max_tenants, pair i = dictionary record i)	0	2	u16 flags: bit 0 present, bits 1–15 zero
  	2	6	reserved, zero
  	8	24	WalOffset
  Slot trailer	0	4	u32 CRC32-C over [0 .. slot_len - 8) of the slot
  	4	4	reserved, zero

  A UUID is its 16 bytes in RFC 4122 order, as the segment header and CHECKPOINT already store them, not a little-endian integer.

  A non-zero reserved byte is a format error, refused at open like a bad checksum. That is the rule that makes the reserved space usable: a future version may define a field there, and a reader of this version must refuse an artefact carrying it rather than parse around it and act on a record whose meaning it does not know. The len = 0 dictionary record is the one place a zero value is meaningful rather than reserved — it marks an unused slot id — and slot id 0 is a usable id, so a dictionary position is never reserved as a sentinel. Because slot ids are u16, max_tenants has a format ceiling of 65,536, refused at config validation with that reason (§3.8).

  The two files share one id space, and that needs saying rather than assuming. RFC 0053’s PUBLISHED carries its own copy of this dictionary — both files must be readable alone — but a copy is not agreement: if each compacted its own tenant list on write, the same tenant would take different ids in the two files and every cross-reading of them would be wrong. So the id space has an owner and a lifetime. RECLAIM owns it: the WAL assigns a tenant its slot id when that tenant is first recorded in either sidecar, in RECLAIM’s dictionary, from the in-memory table open seeds.

  That table is seeded from the union of both dictionaries, not from this file alone, and the difference is not tidiness. RFC 0053’s publish_marks writes PUBLISHED without touching RECLAIM, so a tenant can be assigned an id and recorded only there; seeding from RECLAIM alone would lose that id at the next open and hand it to another tenant, crossing two tenants’ marks in the file that still names the first. So a record in either file — live or tombstoned — reserves its id. A high-water mark backs it: the slot header carries u32 next_slot_id, the lowest id never yet assigned, written by whichever file is written and taken at open as the maximum of the two, so allocation never goes backwards even if one file lags the other by a write. It lives in the four bytes the slot header already reserved at offset 20, so the geometry does not move. That is what makes the ordinary one-file-alone case safe rather than merely usually safe. Ids are never renumbered and never reused while the tenant is recorded in either file — neither file may compact on its own, and a tenant that leaves one but remains in the other keeps its id. PUBLISHED is written from that same in-memory table, so its dictionary is this dictionary, at the same ids, by construction rather than by convention. No operation reassigns an id, not even a rebuild. An earlier draft of this paragraph made the rebuild the one place ids could move; that was wrong, and wrong in a way the resize depends on. A rebuild copies each dictionary record into the wider stride at the same index, so ids are identical before and after — which is exactly what makes a crash between the two files’ renames recoverable, since the half-rebuilt pair still agrees on every id it names. A rebuild that renumbered would have to rewrite every reference in both files atomically, and there is no third witness that could make that crash-safe. A rebuild does still rewrite both files together, from the one in-memory table — that is what keeps their dictionaries the same dictionary across a geometry change — but it copies rather than reassigns, so a pair caught mid-rename comes up at the larger geometry with every id unchanged.

  A tenant introduced by one file alone is the ordinary case, not an error: the two are written at different moments — publish_marks touches PUBLISHED without RECLAIM, a pass touches RECLAIM without PUBLISHED — so at any instant one may name an id the other does not, and the file lacking it simply has no entry for that tenant and reads as “nothing reclaimed” or “no frontier” accordingly. What is fail-closed is narrower: an id that both dictionaries name with different keys, which no correct writer can produce and which silently crosses two tenants’ marks if read past — OpenError::Corrupt, naming both files and the id. A crash between the two writes therefore leaves a readable pair, never a halt.

  Removal follows from the same stability. Dropping a tenant by compacting the dictionary would renumber every id above it, so a removed tenant is tombstoned instead: its dictionary record keeps its key with the tombstoned bit set (bit 0 of the record’s flags field at offset 130), its entries are dropped, and the id is retired for the life of the root — no later tenant takes it, and no rebuild reclaims it, since reclaiming a tombstoned id means renumbering. An operator who exhausts the id space raises max_tenants, which the rebuild path already handles, or starts a fresh root. This design would rather spend ids than invent the witness renumbering would need.

  RECLAIM is the fourth Invariant row in §3.8; the polynomial is RFC 0008 §6.9’s Castagnoli, as everywhere else in the WAL.

• Committed without allocating, because ENOSPC is the case this exists for. A temp-write-and-rename commit needs new blocks, and a full disk is exactly when reclamation must run — writing the record would fail, the pass would unlink nothing, and the WAL would wedge at the one moment the operator needs it to drain. So the record is preallocated once and rewritten in place, never renamed: Wal::open creates (or extends) it to its full two-slot size and fsyncs it, and every pass writes the inactive slot, fsyncs, and the higher generation makes it live — the reader takes the valid slot with the greater generation, so a torn in-place write leaves the previous slot intact and there is no rename, no temp and no new block on the commit path. RECLAIM.tmp therefore does not exist, and §3.2’s temp-sweep selector is unchanged by this record. slot_len is fixed for the life of the file, and that is what makes the two-slot protocol safe: an earlier draft grew the file as the tenant set grew, which moves the second slot’s offset — a crash mid-extension would leave the previously valid slot unparseable at its new position, destroying exactly the torn-write guarantee the two slots exist for. So the slots are sized once, at their bounded maximum, from two configured numbers rather than from the live tenant set: max_unlinks_per_pass (§3.8) and max_tenants, the ceiling RFC 0053 §3.1 introduces with a default of 1024 — the two RFCs land as one pair, and until 0053’s knob exists the implementation uses that same constant. A slot is then 32 + 164 × max_tenants + 24 × max_unlinks_per_pass + 32 × (max_tenants × max_unlinks_per_pass) bytes — the one normative definition of slot_len: its 24 B header and 8 B trailer, a 132 B dictionary record and a 32 B entry per tenant, and max_unlinks_per_pass planned records at a 24 + 32 × max_tenants stride. Fixing those strides changed no arithmetic — the worst case was always what the file reserved — so the figures stand: at the defaults, 1024 tenants and a cap of 128, a slot is 4,365,344 B and the file 8,730,720 B, about 8.3 MiB, next to segments of 128 MiB each. It scales with the product of the two knobs, so an operator who raises both sees it grow; that is the price of a geometry that never moves. Wal::open allocates it once and no later write ever extends it: a tenant set that reaches max_tenants is refused by RFC 0053’s admission guard, not by this file.

  The limit is worth stating plainly rather than claiming more than the design gives: this removes allocation from every pass, not from the WAL’s lifetime. Once the file exists, no pass needs a block, whatever the tenant set does — which is the case that matters, a node whose disk filled reclaiming its way out. A volume that is already full when the WAL is opened cannot allocate it at all, and that failure has nowhere to hide: it happens before any receiver, timer or rotation state exists, so it cannot be “reported” as the rotation path’s terminal class the way an earlier draft claimed. It is its own startup failure — OpenError::Io naming the allocation and the file, the same shape every other open-time I/O failure takes — and the node does not start. The operator frees space and restarts; there is no in-process recovery, because there is no process yet. No design that needs a durable record before unlinking can do better on a volume with no room for the record. RFC0052.17 covers both the at-open failure and the steady-state pass.

• Durable before the unlinks — and off the writer position. The record’s slot write and fsync complete before the pass unlinks anything. The record is on disk before any frame it accounts for is gone; a torn write leaves the previous slot live. None of that file work happens under the journal mutex: serialising every tenant’s entry and three fsyncs is O(tenant count) plus disk latency, which would put fsync time in front of every concurrent append and break RFC0052.12’s O(cap) claim. The pass therefore has two halves. Under the writer position it pops at most the cap’s worth of eligible segments and stale partials from the ledger’s structures — marking each popped entry reclaiming, so it cannot be popped again, while it stays in the ledger and in the byte and segment accounting — and notes the horizon each popped segment is to be reclaimed under per tenant, then releases. Off the writer position it merges those horizons into the record, writes and fsyncs it, and only then unlinks the popped files and fsyncs the parent. The ordering rule is what the startup contract needs and it is unchanged: the record is durable before the segments it covers are gone. What the record says about them must not run ahead of the unlinks, though, so it keeps planned and completed apart: a planned list — one entry per popped segment, its uuid and each tenant’s last offset in it, written before the unlinks — and reclaimed_through, per tenant, which is the proof of loss and is advanced only by housekeeping_commit, after the unlink and the parent fsync succeeded, and made durable by the next record write (the next pass that plans, or the pass’s own second write when nothing is pending). A failed unlink or parent fsync therefore leaves reclaimed_through behind the segment — and makes no promise about the file. An unlink whose parent fsync failed is an uncertain deletion: the entry may or may not be visible after a restart, and this process cannot tell. The file half reports it as ReclaimOutcome::Unlinked { fsync_failed: true }, and since one parent fsync covers every unlink of the pass, the flag marks every path in removed uncertain: housekeeping_commit keeps reclaimed_through behind all of them, marks each segment’s planned entry uncertain, and keeps their bytes in the in-process accounting — the whole batch stays reclaiming; the next pass re-verifies presence — an unlink that finds the file gone completes the reclamation, one that finds it present retries it — and only that verified outcome moves the entry. At open the uncertain entry is reconciled like any planned one (below): present ⇒ retained and re-planned; absent ⇒ the data is gone and is treated exactly like a completed reclamation, reclaimed_through raised, no halt — a planned segment held only covered frames by construction, so its loss is a reclamation that finished, not data lost. An entry leaves the ledger and its accounting only when its deletion is verified, under the writer position at housekeeping_commit (§3.7), and its planned entry is dropped in the same commit. The failure points differ: a failed record write or fsync means nothing was unlinked, so every popped entry is put back to eligible under the writer position and the next pass pops it again — the entry was never out of the ledger, so the no-listing rule costs nothing here; a failed unlink after a durable record leaves the entry reclaiming, still counted, retried by the next pass’s off-lock half within its cap; a failed parent fsync after the unlink is the uncertain case above. Nothing a pass has touched can become undiscoverable, which is what a ledger with no directory scan behind it needs. Housekeeping’s ledger half is the only half that takes the writer position, which is what “runs on rotation’s ownership path” means below.

• Monotonic per tenant, by merge. reclaimed_through only rises: the commit raises each tenant to the highest last-offset among the segments whose unlink succeeded, never lowers an entry, and a tenant absent from it has never had a frame reclaimed. planned entries come and go with their segments. One writer holds the writer position, so the read-modify-write cannot interleave.

• Reconciled at open. A crash between the record write and the commit, or an uncertain deletion, leaves planned entries whose segments may or may not be gone. Open reconciles every planned entry against the directory before anything reads the record: a planned segment still present is retained — it is in the ledger rebuild_ledger() builds, still eligible, and the next pass re-plans it — and its entry is dropped; a planned segment absent is gone, and is treated exactly like a completed reclamation: its per-tenant last offsets raise reclaimed_through as the commit would have, and nothing halts on it, since a planned segment held only covered frames by construction. The reconciled record is written durably before the first pass, and only reclaimed_through is ever read as proof of loss.

• Created at open, and witnessed by CHECKPOINT. The WAL writes an empty record, durably, when it opens a root that has none — before the first pass can run — so the record exists on every root this RFC’s code has ever reclaimed from. Its ordering inside open is part of the contract, not an implementation detail: Wal::open reads the sidecars and only then creates the initial segment, so a record written after that step would leave a first-ever node one crash away from a root holding a segment and neither sidecar — the state the matrix below rejects. The empty record and its parent fsync therefore come before create_fresh_segment, so every directory that has ever held a segment has held a record too, and the fail-closed row can never fire on a node’s own first start. “Missing means pre-RFC” is not safe on its own: a root that has reclaimed and then lost its record (an operator deleting sidecars, a restore from a partial backup) would open, recreate an empty record, and turn a missing snapshot into a pin over frames already gone. So the record has a second witness, one that needs no new file: the CHECKPOINT sidecar’s format version. Presence alone would not do — Wal::checkpoint is unreachable from the pipeline today (#793), but the RFC0008.7 tests and the crash fixture write one, and a root with a version-1 CHECKPOINT and no RECLAIM is a legitimate pre-RFC root, not a lost record. The legacy branch rests on a deployment invariant worth stating, because Wal::housekeeping is public and does unlink under a version-1 checkpoint when called: no production path calls it — the only callers in the tree are ourios-wal’s own tests and the crash fixture (#793 is exactly that finding), so no served root can have reclaimed under version 1. The invariant is belted rather than trusted: on a legacy root, a tenant whose snapshot is missing and whose oldest surviving frame is above its last recorded snapshot horizon fails closed, naming the tenant — the stale-gap shape, which is what reclamation under a version-1 checkpoint would leave behind — so a hand-built or test-derived root that did reclaim is caught rather than silently pinned. So this RFC bumps the sidecar’s VERSION field (the u16 at bytes 4–6 of the 32-byte OWCK record, 1 today) to 2, and the read path accepts both: a version-1 CHECKPOINT with RECLAIM absent is a pre-RFC root — nothing was ever reclaimed on it (#793) — opened without creating a record, and its CHECKPOINT is rewritten at version 2 by the first checkpoint, the record being created (empty, armed) on that same path. That upgrade is version-aware, not mark-aware: Wal::checkpoint returns Ok(()) without writing when the offered mark equals the current one, which on an idle node is every barrier after the first, so a root whose first barrier stamps an unchanged mark would never reach version 2 and housekeeping would stay skipped forever. So an equal mark still rewrites the sidecar — and arms the record — whenever the on-disk version is 1; only an equal mark on an already-version-2 sidecar keeps the no-write fast path. Creating it eagerly at open would leave a state the matrix cannot read — a version-1 CHECKPOINT beside an unarmed RECLAIM — for as long as the node runs before its first barrier; deferring the creation to the upgrade keeps a restart in that window on the legacy branch, which is exactly where it belongs. “Created at open” below therefore means on a root with no version-1 checkpoint to migrate. A version-2 CHECKPOINT with RECLAIM absent is a post-RFC root that has lost its record — every unlink is gated on a checkpoint, and on a post-RFC root the record is created at open, before the first checkpoint can exist — and open fails closed on it as OpenError::Corrupt, naming the missing file; both absent is the pre-RFC layout with no checkpoint — and that is the shape every live pre-RFC root has, since Wal::checkpoint has no production caller (#793): a served node today holds segments and neither sidecar. An earlier draft of this row failed such a root closed, which would have refused to start every existing deployment; the rule is therefore scoped to roots that carry a post-RFC witness — and the witness has to be one every post-RFC root has, which a snapshot artefact is not: a NoConsumer root (§3.2) writes no snapshots at all, so keying on them would read a WAL-only root that lost both sidecars as legacy and recreate an empty record over real reclaimed-through proof. The witness is therefore the segment header’s format version, which every root with any segment necessarily carries: SEGMENT_VERSION is bumped from 1 to 2 for segments created under this RFC, and the reader accepts both — version 1 is a segment written before this RFC, version 2 one written after. A root can be upgraded without ever rotating, though — the checkpoint arrives on a barrier, rotation on an append — so the witness cannot wait for a segment: the first pass that reclaims writes the version marker before it unlinks anything. In practice that is the record itself, which §3.2 already requires to be durable before the first unlink and which no pre-RFC root has; the segment version is the witness that survives when the record does not, and a root that reclaimed before it ever rotated is covered because its reclaim wrote the record first. If an implementation finds a root that can reclaim with neither a version-2 segment nor a record, that is a bug in the ordering, not a case for the matrix to absorb. A visible CHECKPOINT is not yet a durable one: its rename’s parent fsync can fail after the entry is visible to this process, exactly as §3.2 says for the snapshots root, so startup revalidates the WAL root the same way — one directory fsync before the sidecars are read, which makes every entry the listing saw durable — and fails startup when that fsync fails, as a RecoveryDriverError, rather than trusting a version-2 sidecar that may not survive the next crash. The same fsync covers RECLAIM, which lives in the same directory. The ordering rule that makes this witness safe is the one the fresh-root case already established, generalised: no version-2 segment is ever created before the record is durable. A legacy root rotates before it ever checkpoints — rotation is append-driven and the first barrier may be minutes away — so a first rotation that installed a version-2 segment while the root still had no sidecar would leave exactly the shape this row rejects, and the node would refuse to open after a restart: a live root bricked by rotating. So the first rotation on a legacy root writes the record (armed, mode unrecorded) and fsyncs it before it creates the fresh segment, the same order Wal::open uses on a fresh root. A root holding any version-2 segment with both sidecars absent has run under this RFC and its missing sidecars are a loss — OpenError::Corrupt, naming both files, because reading it as “fresh” would replay frames whose Parquet rows may already exist. Only version-1 segments and neither sidecar is the legacy shape and takes the legacy branch, belted by the same stale-gap check every legacy root gets (a tenant whose snapshot is missing and whose oldest surviving frame is above the horizon its snapshot records fails closed). That check needs a horizon on a root whose snapshots are version 1, which §3.1 discards as unknown-version — so for this check only, the legacy artefact’s global wal_high_water is decoded: it is a mark the old writer stamped only after every tenant’s frames at or below it were drained and mined, which makes it a sound lower bound on what that tenant had folded, and reading it here restores nothing and feeds no miner. A snapshot that cannot be decoded even for that field, or is absent entirely, has no horizon to compare and fails closed naming the tenant rather than being read as “nothing reclaimed” — the one direction that could replay past reclaimed data. Accepting both versions is the whole migration, as with the other two bumps, and a fresh root of either mode stays bootable because its record is durable before its first segment exists. A fresh root under this RFC never reaches that row at all: its record is written before the first segment is created (the bullet above). Per the pre-production layout policy that read path is the whole migration — no tooling — and the implementing PR carries the ! marker for the sidecar version bump beside the snapshot one. The matrix is symmetric: a RECLAIM with any reclaimed_through or planned entry beside a missing CHECKPOINT is also fail-closed, naming both files — every unlink was gated on a checkpoint, so the checkpoint existed and is gone, and without X the Parquet-side suppression horizon cannot be rebuilt and replay would republish. An empty record is not enough to tell the other way: a post-RFC node can checkpoint many times while reclaiming nothing, and deleting that checkpoint would otherwise read as a fresh root and replay published rows. So the record header carries a two-state witness, because one flag written before the checkpoint would make the window between that write and the checkpoint’s rename read as a lost checkpoint — a crash there is ordinary, not corruption. checkpoint_armed is written, durably, before the first version-2 CHECKPOINT attempt; checkpoint_seen is written at the next record write after that checkpoint succeeded (the next pass’s write, or an immediate one when no pass is pending), and once seen the flag never clears. The matrix reads them in that order: seen with CHECKPOINT absent is a lost checkpoint and fails closed, naming both files, whatever the entries say; armed without seen and CHECKPOINT present at version 2 is the ordinary post-upgrade state — the checkpoint is durable, the crash simply landed before the record’s next write — so open promotes it, setting seen durably before anything reads the matrix, and it is never a fault; armed without seen and CHECKPOINT still at version 1 is the migration’s own retry state, which this ordering creates when the upgrade write fails between the arming and the rename — the record is armed, no version-2 checkpoint exists, and nothing can have been reclaimed, since housekeeping is a no-op until the witness exists — so the root opens normally on the legacy branch and the next checkpoint retries the upgrade, re-using the arming already on disk; armed without seen and CHECKPOINT absent splits on whether the directory holds segments. With no segments it is a genuinely fresh root whose arming preceded a checkpoint that never landed: the record is opened empty and re-armed by the next attempt. With segments present it is not fresh at all — the first rotation on a legacy root writes the record before it creates its version-2 segment (below), so this is that root, mid migration, with the upgrade still owed — and the state is retained, not reset: the record keeps its arming and its mode, the root opens on the legacy branch, and the next checkpoint retries the upgrade. Emptying the record there would discard a mode the first pass may already have adopted; neither flag with CHECKPOINT absent is the post-RFC root before its first barrier, which is what open creates. Entries and seen cannot disagree, since every unlink is gated on a checkpoint that must have succeeded to produce them. RFC 0053 adds PUBLISHED to this matrix on the same footing: its §3.2 makes a missing PUBLISHED beside a version-2 CHECKPOINT fail-closed like a missing RECLAIM.

• Corrupt is fatal. The record carries a version byte and a checksum; a record failing either at open fails startup as OpenError::Corrupt, naming the file. It is not read as missing: missing means “never reclaimed”, damaged means “reclaimed, extent unknown”, and only the first is safe to proceed from.

NoConsumer means no miner state exists, and that is a precondition, not a hint. The previous draft read a NoConsumer reclaim as “checkpoint-covered” and let a later start pin instead of halting. That is unsafe wherever a miner is running: hazard #5’s retain rule (RFC 0001 §6.9) makes a missing snapshot survivable because the WAL still holds the frames to re-mine, and a pass that reclaimed by checkpoint alone has removed exactly those frames — the tree cannot be rebuilt, and wiring snapshots up afterwards does not bring it back. Nothing in the code prevents that combination today: Wal::housekeeping’s retain_floor (what SnapshotHorizons replaces) is a plain Option, its only None callers are ourios-wal’s own tests, and the one WAL-bearing role the server ships — the ingester — always holds a MinerCluster.

So the mode is given a precondition rather than a reading. NoConsumer means the caller holds no miner state at all: a WAL used without a miner (the crate’s own tests and benches today, a WAL-only role if one is ever added). The ingest receiver is never in that position and therefore always passes Known — a tenant with no valid snapshot is expressed by the Pinned floor §3.2 already defines, never by NoConsumer, which is the distinction RFC0052.13 exists to hold. The precondition is asserted where SnapshotHorizons is built, and the WAL enforces it from durable state rather than from inference: the RECLAIM header records the root’s consumer mode, and any pass whose mode disagrees with the recorded one is refused as a ReclaimError naming both modes, before anything is planned or unlinked. An earlier draft refused only a NoConsumer pass on a root that already held a Known entry, which is not fail-closed: a miner-bearing root that has checkpointed but never reclaimed has no entry at all, so the first mistaken NoConsumer pass would pass the check and delete frames the miner needs. The header does not depend on an entry existing. It does depend on knowing the mode, and Wal::open does not: the mode arrives with SnapshotHorizons at housekeeping_prepare and nowhere else, and threading it into WalConfig would put a receiver-side fact into the WAL’s construction — where a mismatch would have to be an open-time error on a root that has not yet reclaimed anything. So the record is created with the mode unrecorded, and the first pass adopts its own mode and persists it durably before unlinking anything. The crash matrix for that window is one row: a root that dies between the adoption write and the first unlink comes back with a mode, no entries and every segment present, which is exactly a first-pass-never-ran root and is read as such. §3.7’s surface list matches — no mode on Wal::open, the mode on housekeeping_prepare’s horizons — and the first reclamation on any root is already governed by a durable mode.

Entries still record their mode too, and recovery reads them by it: an entry written under Known whose tenant’s restorable horizon is below it — including a tenant with an entry and no restorable snapshot at all — is unrecoverable state and recovery halts, naming the tenant; a NoConsumer entry is checkpoint-covered, which is sound because the header’s mode made every pass on that root a no-miner pass. A tenant with no entry has lost nothing and pins at its oldest surviving frame. An undecodable or missing snapshot is therefore safe to fall back from exactly when the record proves it is, and RFC0052.17 holds each of those cases.

An entry whose tenant has no surviving frame is satisfied by absence. A tenant leaves the ledger when its last surviving segment is unlinked, but its reclaimed_through entry stays in the record; were the entry still pending, every later Known pass would need a restorable horizon for a tenant with nothing left in the WAL, and one lost snapshot after ordinary churn would halt reclamation for every other tenant on the root. The entry exists so that recovery never rebuilds a tenant’s miner state from a log missing its oldest frames; with no surviving frame there is no log to rebuild from and nothing a pass could unlink on that tenant’s behalf, so the check above applies only to a tenant the ledger holds. The ledger is the witness, since rebuild_ledger() rebuilds it from every surviving frame’s prefix at the end of recovery and every live append updates it — and only once that walk has run: before it, the ledger knows only frames appended since open, a tenant missing from it proves nothing, and the entry stays pending. The entry is not removed: absence is not retirement — “no surviving frames” is not “will never write again” — and the WAL is the wrong place to decide the latter. The dictionary tombstone stays in the format for a caller that can decide it; nothing in this RFC writes one.

A churned-out tenant that writes again halts reclamation until its first new snapshot installs. Its new frames put it back in the ledger, so the entry is pending once more, and with its snapshot lost there is no restorable horizon to satisfy it: the pass fails closed with HousekeepingError::Unrecoverable naming the tenant. That is intended. Treating the return as a fresh tenant would discard the entry that makes a missing snapshot loud, and a later restart would rebuild that tenant’s miner state from frames that begin after the ones already reclaimed, silently. The pause lasts until the tenant’s next snapshot installs and a pass runs — at default tunables about one barrier_secs (300 s) plus one housekeeping_secs (60 s) — and only reclamation pauses: ingest, acks and queries are untouched. If that tenant’s snapshots keep failing to install, the pause and the WAL’s growth last as long as the failures do; every pass meanwhile names the tenant, so the stall is loud rather than silent.

The floor is also the reason §3.1 can tolerate a failed snapshot write. housekeeping reclaims only segments every tenant’s horizon covers, and never past the checkpoint, so a stale floor makes truncation conservative — it retains frames a snapshot has not captured, which degrades the next start to a fuller replay and never to loss (hazard #5’s retain rule, RFC 0001 §6.9).

Housekeeping’s ledger half takes the WAL’s single-writer position, so it runs on the same ownership path as rotation rather than concurrently with it; its file half — the record write, the unlinks, the parent fsync — runs after that position is released, on the housekeeping task alone.

3.3 Rotation failure becomes recoverable, under a bounded retry

A quiesce must stop being permanent. The five sites — the four the code has today and the rename(partial, final) step the temporary name adds — are not equally retryable, and the design treats them by what they leave behind:

The last row is the one to read carefully. Because the rename happens before that fsync and the install happens before it too, there is no orphan to clean up there — the file is the live append target. Unlinking it would destroy the segment the WAL is currently writing to. The retry is the directory fsync alone, carried by dir_fsync_pending.

One rotation obligation at a time. A Rotation-origin dir_fsync_pending outlives the rotation that created it, and a later append or timer tick can find the segment due for rotation again. Starting that second rotation would install a segment whose predecessor’s directory entry is still not durable, and a crash there loses the ordering the first obligation exists to restore. So any path that begins a rotation — an append that observes the size or age trigger, and Journal::rotate of either kind — first discharges a pending Rotation obligation, and refuses the rotation if the discharge fails. The refusal is not a new class: it consumes one unit of the retry budget like a failed rotate, so it is reported transient while the budget holds and server-terminal, client-retryable once it is exhausted (§3.3’s classification, RFC0052.15) — and no batch is acknowledged in either case. An Open-origin obligation does not gate a rotation: it is an ordinary retryable sync failure outside the budget, and the rotation’s own parent fsync discharges it.

The orphan is why the current code quiesces rather than retrying, and the existing comment says so: a surviving directory entry whose header bytes were lost fails the next Wal::open’s header read, turning a benign crash into OpenError::Corrupt.

Unlinking the orphan before each retry is not sufficient, and this RFC does not rely on it. A remove_file is not durable until the parent directory is fsynced, so a crash in between can bring the entry back; and after the final failed retry a header-only orphan under the final name can still be sitting there, which Wal::open may select as the newest segment — that is the legacy shape RFC0052.11 handles; a .wal.partial is never selectable, since list_segments returns only *.wal. Either case contradicts RFC0052.4/.5’s restart property, so the design removes the orphan class instead of cleaning up after it:

A fresh segment is created under a temporary name, its header fsynced, then renamed to its final .wal name, and only then is the parent directory fsynced. The ordering matters in both directions:

• the header must be durable before the rename, or a surviving .wal entry can have unreadable header bytes — today’s hazard;
• the parent fsync must come after the rename, because the rename creates the directory entry and an fsync taken before it does not make that entry durable.

And the new segment is installed before that fsync, not after. This is what makes the failure path reachable rather than a description of a state the code cannot be in: rotate sets current_segment/path/uuid to the renamed file first, then fsyncs the parent, and on failure sets dir_fsync_pending and returns without quiescing. The new segment is then genuinely current, so sync — which fsyncs current_segment and discharges dir_fsync_pending — closes the obligation, exactly as it already does for the segment open creates. Since no batch is acked until sync returns Ok, no frame is ever acked in a segment whose directory entry is not durable.

The retry is the group-commit sync, not the next append, and saying “a retry re-attempts the fsync” blurred that. Once the new empty segment is current, the next append’s rotation check is false, so it simply writes its frame; nothing re-enters rotate. What retries is the sync that follows, which is the right place: it is the operation acks are gated on, so a repeated directory-fsync failure repeatedly refuses to ack rather than accumulating unacked frames behind a rotation that never reruns. Those appends are not lost either — they are in the segment, and the next successful sync covers them.

That retry draws on the same budget and reaches the same terminal state — when the obligation came from a rotation. dir_fsync_pending starts true on every Wal::open, fresh or reopened, so the flag alone cannot say where the obligation came from; it therefore carries its origin, Open or Rotation. A failed discharge of an Open obligation is an ordinary retryable sync failure outside the budget, exactly as today. A failed discharge of a Rotation obligation consumes one unit of the rotation retry budget, exactly as a failed rotate does, and exhausting it enters the terminal state whichever operation gets there — which is what keeps RFC0052.15’s narrowness true: an ordinary fsync failure never becomes terminal. Both AppendError and SyncError carry a typed terminal variant, and IngestFailure::classify maps both under the terminal-only rule. The plumbing through group commit is stated too, because today it would lose the type: CommitCoordinator::flush collapses every sync error into SyncFailure { detail: String } and waiters rebuild a generic ReceiveError::WalSync. FlushOutcome therefore gains the error’s class — terminal or transient — beside the detail, and waiters rebuild a ReceiveError that carries it, so the classifier sees the terminal state rather than a string; §6 drives both a rotation-origin terminal sync failure and an ordinary transient one through flush and the waiter, so the production path cannot erase the class while the classifier tests pass. The retry hint is settled honestly: an append, sync or still-retrying rotation failure has no scheduled retry on the server — the next attempt is the client’s next request — so those transient classes carry no hint and the client backs off as OTLP prescribes, exactly as the merged #794 left them; a hint appears only where a real server-side cadence exists, which in stage 1 is nowhere and in RFC 0053 is the backpressure state’s housekeeping cadence. The transient/terminal distinction therefore rides the classification and the Status message, not a header — so a persistent parent-directory fsync failure is not left as an ordinary WalSync that RFC0052.5 and RFC0052.15 could never observe.

Installing after the fsync, as the draft implied, leaves the renamed file owned by nobody: rotate would still point at the old segment while a complete .wal sits beside it, which is the orphan case all over again.

A rotation that fails before the rename therefore never leaves a file that looks like a segment. This amends RFC 0008 §6.5 explicitly, and the amendment is wider than the create sequence: §6.5 required the new segment’s directory entry to be durable before any frame landed and a permanent refusal after a rotation failure, and Scenario RFC0008.6 asserts both. Under this RFC the segment is installed before its parent fsync, frames may land while dir_fsync_pending is retried (none acked until sync discharges it), and the refusal is bounded rather than permanent — so RFC0008.6 is superseded by RFC0052.4/.5 and §6.5’s durable-entry-first wording by §3.3 here. It is the one on-disk behaviour change in this RFC; it needs no format or schema change, because the temporary name never becomes a segment.

The post-rename window is not covered by that, and needs its own rule. If the final sync_parent_dir fails, the .wal name already exists — so the “never looks like a segment” property does not hold there, and the temporary -name sweep does not reach it. That file is a complete, header-durable segment whose directory entry may not survive a crash, which is exactly the state dir_fsync_pending already exists to describe. So it is kept, not unlinked, and the rotation records that the directory fsync is still owed: the next sync discharges it before acking anything, as it already does for the segment open creates. A retry after this failure re-attempts the fsync rather than creating a second segment.

Temporary files are swept explicitly, not implicitly. list_segments returns only *.wal, so a temporary file is invisible to it — which is the point for Wal::open, and means housekeeping would not remove it either. Left at that, a persistently failing rotation would leave one temporary file per retry, consuming disk that reclamation exists to free. So housekeeping gains a second, explicit step — but “temporary files” is far too broad a selector to state loosely. The WAL root already holds CHECKPOINT.tmp and the snapshots directory holds *.snap.tmp; a sweep matching *.tmp could delete an in-progress checkpoint or snapshot, which is a worse failure than the debris it collects.

So the name is reserved and exact: the in-progress segment is <uuid>.wal.partial, and every file of that shape in the WAL root belongs to the WAL — the sweep cannot tell rotation debris from a file an operator placed under a reserved name, so it does not try, and unlinks any stale one. A file that does not match the shape (foo.wal.partial, anything else) is ignored, as non-segment files are everywhere else in the WAL, and *.tmp is never matched: it stays the checkpoint and snapshot namespace. Two further rules make it safe:

• there is no in-flight partial to protect: housekeeping takes the single-writer position, so no rotation attempt is in progress while it runs, and every partial it sees is debris — an earlier draft’s name-based “skip the live one” rule had no ownership path and is withdrawn;
• the unlinks are followed by a parent-directory fsync, as housekeeping’s segment unlinks already are, so a crash cannot resurrect swept debris.

The step runs inside the same per-pass cap as segment unlinking (§3.7) and counts against it, because it pops from the same ledger under the same single-writer position — a backlog of stale partials would otherwise make a “bounded” pass do unbounded directory work and break RFC0052.12; the unlink itself runs in the pass’s off-lock half like a segment’s (§3.2). Within the cap, partials are popped first and segments take what remains: a partial is bounded per rotation by the retry budget and holds no frame, while a segment backlog is unbounded, so the other order would let the incident’s backlog consume every pass’s cap and leave rotation debris on disk for as long as the backlog lasts — contradicting RFC0052.4’s “one rotation’s debris clears in one pass”. An unlink failure is logged and retried on the next pass like any other.

Two more properties of the sweep are stated because the current housekeeping has neither. It runs on every pass, regardless of the checkpoint precondition: today housekeeping returns at once when no checkpoint exists, and a rotation that fails before the first checkpoint would otherwise leave partials that every later pass skips; only the segment-unlink portion is gated on a checkpoint. And it lists nothing on the pass: partials left by a previous process are seeded into an in-memory partial list by rebuild_ledger(), which walks the directory at the end of recovery anyway, live rotations register the partial they create, and the sweep pops from that list under the same cap — so restart debris is found without a directory scan under the writer lock. Its error path still makes completed removals durable: when a pass fails part-way — a header read, a stat, a later unlink — the parent-directory fsync covers every unlink that already succeeded before the error is returned, and HousekeepingProgress reports the partial count. Today’s single fsync after the complete scan would let a crash resurrect a partial the pass had already removed.

Pre-existing *.wal orphans need their own answer, and neither changing future rotations nor guessing at open is it. A node wedged before this RFC lands may already hold a segment whose header fsync failed, so its header bytes may be partial — Wal::open validates the newest segment’s header and can return OpenError::Corrupt on it, while older segments are checked by Wal::replay and surface as recovery errors; either path halts. Claiming these are “real zero-frame segments” was wrong.

An earlier draft of this section had Wal::open unlink such a file when it was the newest and the only unreadable one. That is withdrawn. An unreadable newest segment is indistinguishable from genuine header corruption, and RFC0008.5’s contract is that corruption halts rather than being cleaned up; a heuristic that unlinks it can silently discard real data, which is a worse failure than the one it avoids. No shape-based guess is safe here, because the shape carries no evidence of which cause produced it.

So: these nodes keep today’s behaviour and still halt at open. What this RFC adds is that the halt is actionable — the error names the file, describes the observable shape (newest segment, unreadable header) and notes that a failed rotation is one way to produce it, without claiming that it did, so an operator can weigh removing it. Whether that deserves a WAL verb rather than a documented manual step is in §7; either way the decision is a human’s, because only a human can weigh “this is probably rotation debris” against “this might be my data”. Future rotations cannot reach this state at all, so the population is finite and shrinking.

A pre-rename retry is attempted on the next append, and a post-rename directory-fsync retry by the next sync (above); neither runs in a background loop of its own: the WAL is single-writer and has no task, and a rotation is only needed when there is something to write. The one append-independent caller is the barrier task’s idle rotation (§3.2, Journal::rotate(Discretionary)), and it is not exempt: a timer-triggered rotation that fails draws on the same budget and can reach the terminal state without an append ever arriving, which RFC0052.4/.5 cover. quiesced becomes a typed state carrying the attempt count and the first underlying error, so the distinction between “retrying” and “given up” is representable rather than a bool. The budget is a count: three consecutive failed attempts by default (rotation_retry_attempts, beside the other rotation knobs), counting a failed rotate and a failed Rotation-origin fsync discharge alike, reset to zero only when the pending obligation itself succeeds — the rotation, or the directory-fsync discharge — never by an unrelated successful operation, or a parent fsync failing behind succeeding data syncs would never reach the terminal state; with no backoff of its own — each retry rides the next append or sync, whose cadence is the backoff. After the budget is exhausted the WAL enters a terminal state that is still reported distinctly (see §3.5) and still refuses appends — a disk that has failed the same fsync a dozen times is not going to be fixed by a thirteenth attempt, and hammering it obscures the real fault.

The retry must not widen the ack surface: a rotation that has not completed means the append did not happen, so the batch is not acked, exactly as today.

This is a contract change against a passing test, and says so. rfc0008_6_rotation_failure_quiesces_the_wal asserts that after a rotation failure “every subsequent append is refused … even after the underlying condition clears”. That is precisely the behaviour §3.3 replaces, so the test is not inconvenient — it is the specification of what we are choosing to change. Per CLAUDE.md §6.2 the implementation must surface that explicitly and get approval before editing it, rather than quietly adjusting it to pass. The replacement asserts the new contract on both sides: a transient failure recovers (RFC0052.4) and a persistent one still refuses forever (RFC0052.5), so the protection the original test provides is kept, not dropped.

This RFC amends RFC 0018 §3.2, and that amendment belongs here, with the retry it depends on. §3.2’s transient class lists “post-rotation quiesce” alongside WAL append I/O and fsync failures, all mapping to UNAVAILABLE / 503 “with an optional Retry-After header”. Classifying the quiesce as transient was right when the latch was assumed to be a momentary condition; #791 showed it is permanent until a restart, so the classification is wrong in the one way that matters — it tells a client the node will recover on its own.

The amendment is narrow and does not touch §3.2’s binding rule, which is that a transient failure must never carry a non-retryable code. The status stays UNAVAILABLE / 503 for exactly the reason §3.2 gives: the batch was not acked, and every non-retryable OTLP status also tells the client to drop it. OTLP’s retryable axis is about the data — whether the client may resend the same batch — and here it may and must: the batch is valid and will be accepted once an operator has cleared the node, so a non-retryable mapping would be wrong on the spec’s own terms. What changes is the class, and §3.2’s two classes cannot say it: the state is neither transient (no delay fixes it) nor permanent in §3.2’s sense (nothing is wrong with the batch). So this RFC adds a third class to §3.2’s table — server-terminal, client-retryable: the node needs an operator; the client keeps its data and retries with exponential backoff, which is what OTLP prescribes when no Retry-After / RetryInfo is sent, and none is, because the server cannot predict when the state clears. Under this section’s bounded retry, only the terminal state enters that class. A rotation failure still within its retry budget genuinely is transient: a later append can succeed, so it stays in the transient class and its message says it is retrying. Only once the budget is exhausted is the node in a state no delay fixes, and only there does the classification change, with the message naming the state. No retry hint is carried on either — the server schedules no retry of its own (§3.7) — and RFC0052.15 pins both halves on class and message.

3.4 Backpressure — moved to RFC 0053

The explicit local bound, its pre-append reservation, its Retry-After contract and its livelock fix are RFC 0053 §3.1. They depend on this RFC’s timer (§3.1), barrier exclusion (§3.1) and ReclaimState (§3.7), which is why they follow it rather than share a document with it. The number is kept so the review history’s section references still resolve.

3.5 The state becomes observable

WalMetrics already carries unflushed_bytes, disk_bytes and segment_count, and none of the three is exported: the only WAL metric names in the registry are ourios.wal.append.duration and ourios.receiver.wal.truncated. unflushed_bytes is read only internally, for the fsync-batching decision. So “the WAL stopped growing” — the incident’s one observable symptom — was unobservable even though the numbers existed in memory.

This RFC exports the existing three and adds the state the other sections actually depend on. ReclaimState (§3.7) carries, and the exporter surfaces:

• unreclaimed bytes and the age of the oldest unreclaimed frame — the bytes are what RFC 0053’s bound is measured on, and the age is what an operator alerts on; “all unreclaimed” means including the post-checkpoint tail, the current segment and anything the floor retains. No frame carries a timestamp, so the age is a defined proxy: the oldest surviving segment’s UUIDv7 timestamp, its creation time, which is older than or equal to every frame in it and therefore conservative — and None whenever unreclaimed frame bytes are zero, since the header-only current segment Wal::open creates or retains holds no frame and an idle WAL must not report a growing age. An earlier draft said “below the checkpoint”, which would have exported the one number that does not grow during an outage;
• the retain floor and its lag — lag_bytes, the frame bytes in the segments at least one tenant still holds back, less the current append segment, and lag_segments, their count; both are reported as zero while the floor is Unknown (no pass has derived one, so no lag is reported yet, whatever the ledger holds) or None (the checkpoint alone governs, so no tenant holds anything back). The figures are conservative by construction: equal to the bytes and segments the floor retains below the checkpoint whenever the checkpoint covers every held segment, larger by the held post-checkpoint tail otherwise, never smaller. An earlier revision bounded them by the checkpoint, which made the held set move whenever the checkpoint does — a range count on every advance, the eager promotion RFC0052.12 forbids — and a retention gauge errs high safely. The current segment is excluded for the reason unlink_remaining excludes it (§3.7): no pass can pop it, and counting it would hold the gauge above zero on a healthy node. WalOffset is a (UUIDv7, byte) pair and has no subtraction of its own, and the figures come from per-segment frame-byte accounting the WAL keeps incrementally (seeded by rebuild_ledger(), updated on every append and unlink) — never from an inspection, which §3.7’s cap would bound and a large backlog would make untrustworthy — including whether it is RetainFloor::Pinned and by how many tenants — without which RFC0052.13’s “an operator can tell a lagging tenant from an unexplained stall” is not achievable, since that is the case where reclamation stops with a healthy store;
• the rotation-failure state: retrying, with its attempt count, versus terminal;
• the housekeeping backlog — horizon_remaining and unlink_remaining from the last pass’s HousekeepingProgress (§3.7), exported as two asynchronous gauges in {segment}: the proposed names are ourios.wal.housekeeping.horizon.remaining and ourios.wal.housekeeping.unlink.remaining — dotted namespace under the existing ourios.wal.*, snake_case components, not pluralised, no unit in the name — with the registry bump settling the final spelling. They are what turns capped from a bool into a trend: a backlog that is draining and one that is growing read alike on capped;
• the cadence_failed latch — pipeline-owned rather than a ReclaimState field, because it is set by the ingester’s unwind guards (§3.1: the encode pool’s BatchGuard, the sink’s publish guard, the barrier’s own step) and no WAL call observes it; exported as the failed epoch (absent when clear) beside the current barrier epoch, so an operator can see that the latch is set and which cuts it fails; the receiver exports it beside the ReclaimState fields, and RFC0052.7 names which surface each item comes from.

A log event is emitted on entering and on leaving a refusing state — for transitions observable within one process; the terminal rotation state’s only exit today is a restart a fresh WAL cannot observe, so it has an entry event and no leave event until §7’s operator verb exists to emit one (RFC0052.7 says the same). Names come from the shared ourios-semconv registry in one bump, not hand-written, and error.type continues to carry the failure class on existing counters rather than spawning per-error metrics.

Transitions have an observer, so “exactly one event” is implementable. A state snapshot alone cannot emit anything. The receiver’s timer task is the single emission owner for the states that change on ticks: it compares a categorical projection of ReclaimState — floor kind and latch, never the byte, age or lag figures, which change on every pass — before and after each maintain call and emits on change (floor pinned or lifted, latch set). Rotation-state transitions are not sampled that way, because a rotation can fail on one append and recover on the next before either sample and the sampler would see no change: the coordinator emits those edges synchronously at the transition, under the journal mutex, as RFC 0053 does for its refusal latch, so entry into Retrying or Terminal and the recovery out of Retrying each emit exactly once at the point they happen. The channel is defined: the coordinator records the WAL’s rotation state after every append and sync outcome into a shared IngestState cell — written by the WAL itself under its own mutex, in the order it observed the outcomes, never from flush outcomes that publish monotonically by covered_seq rather than by completion, so an older transient outcome cannot overwrite a newer terminal one — and the projection is built from that cell, from ReclaimState, and from the task’s own cadence_failed latch — an explicit input, not a ReclaimState field. Rotation-state edges, including the terminal entry, are emitted synchronously at the transition as stated above — never sampled, never late. One owner per state, real call sites, registry names.

3.6 Requeue on unwind — moved to RFC 0053

The duplicate-versus-lose decision, the consuming-call ownership rule and the age-sweep survival that follows from them are RFC 0053 §3.2. They are deferred rather than dropped: #795’s stop-on-panic plus §3.1’s cadence_failed latch is the safe interim, and it is what RFC 0053 replaces. The number is kept so the review history’s section references still resolve.

3.7 Ownership and API surface

§3.1 and §3.2 are not callable as written today, and saying so precisely is part of this RFC’s job.

The barrier has no WAL handle. flush_then_snapshot receives the record sink, the audit sink, the snapshots root, the miner, an Option<WalOffset> high-water mark and a cadence label. The WAL itself is boxed as a Box<dyn Journal> inside a Mutex in the commit coordinator, which is where the single-writer position lives (RFC 0008 §3.1).

Journal has no reclamation surface. The trait is append_batch, sync, unflushed_bytes. Wal::checkpoint and Wal::housekeeping are inherent methods on the concrete type, so the pipeline cannot reach them through the trait object.

So the design is:

• Journal gains the two reclamation methods and a state accessor, with a single object-safe error type rather than the concrete WAL’s two:

  enum ReclaimError {
      Checkpoint(..),
      Housekeeping { progress: HousekeepingProgress, source: .. },
  }                                                        // object-safe, one type
  struct HousekeepingProgress {
      removed_segments: usize,   // what the forced-rotation trigger reads (RFC 0053)
      removed_partials: usize,
      capped: bool,              // "more to do" versus "backlog drained"
      horizon_remaining: usize,  // segments a received horizon may not yet have
                                 // been applied to: per tenant, the segments it
                                 // has frames in above its cursor, summed over
                                 // tenants (a segment counts once per tenant behind
                                 // on it) — a counter the ledger keeps INCREMENTALLY,
                                 // raised when a tenant's frame first lands in a
                                 // segment above its cursor and lowered as the
                                 // cursor moves, so reading it is O(1) under the
                                 // guard, never a per-tenant sum on the pass, and a
                                 // rising horizon costs no range count. Conservative:
                                 // equal to the cursor-to-horizon span whenever each
                                 // tenant's horizon covers its newest segment,
                                 // larger otherwise, never smaller; zero before
                                 // the first Known pass (an empty Known map
                                 // counts as one) and under NoConsumer
      unlink_remaining: usize,   // empty-set segments not yet popped (waiting on the
                                 // checkpoint or on a pass — counting only those at
                                 // or below the mark would cost a range count on
                                 // every checkpoint advance, which §3.7 keeps O(1))
                                 // plus entries reclaiming or uncertain — likewise an
                                 // incremental counter, moved when a set empties and
                                 // on every entry state change (eligible, reclaiming,
                                 // uncertain, removed), O(1) to read.
                                 // Both are the values as of the END of the ledger
                                 // half that produced this struct (prepare's plan
                                 // or commit's return), never re-derived off-lock
      floor: RetainFloor,        // derived on this pass (§3.2)
      lag_bytes: u64,            // §3.5's units, from per-segment accounting —
      lag_segments: usize,       // never from an inspection the cap would bound
  }                              // carried on Err too, so partial work, floor and
                                 // lag stay observable on the failure path
  
  /// Why a floor is or is not available — `Option` cannot carry this.
  enum RetainFloor {
      Unknown,               // no pass has derived it yet: reclaim nothing, report as such
      None,                  // no snapshot consumer exists: checkpoint alone governs
      Min(WalOffset),        // the minimum over every tenant's horizon
      Pinned { offset: WalOffset, tenants: usize },
                             // some tenant has no valid snapshot: the minimum
                             // includes its oldest surviving frame
  }
  
  fn append_batch(&mut self, payload: &[u8]) -> Result<WalOffset, ReceiveError>
                                                           // the ReceiveError boundary the
                                                           // coordinator and both transports
                                                           // classify against is kept; only
                                                           // the offset Wal::append already
                                                           // produces is added (§3.1's mark)
  fn checkpoint(&mut self, durable_to: WalOffset) -> Result<(), ReclaimError>
  enum SnapshotHorizons { NoConsumer, Known(HashMap<TenantId, WalOffset>) }
  fn housekeeping_prepare(&mut self, horizons: &SnapshotHorizons, max_unlinks: usize)
      -> Result<ReclaimPlan, ReclaimError>             // the ledger half, under the guard:
                                                       // applies horizons (capped), derives
                                                       // RetainFloor, pops partials then
                                                       // segments, marks them reclaiming
  struct ReclaimPlan {                                 // owned paths: the file half needs
      segments: Vec<PlannedUnlink>,                    // no guard and no WAL handle
      partials: Vec<PathBuf>,
      record: ReclaimRecord,                           // the merged per-tenant horizons
      root: PathBuf,                                   // for the RECLAIM slot write and the
                                                       // parent fsync after the unlinks
      progress: HousekeepingProgress,                  // floor, lag, capped — as of prepare
  }
  enum ReclaimOutcome {                                // what the file half did
      RecordFailed(io::Error),                         // nothing unlinked
      Unlinked {
          removed: Vec<PathBuf>,                       // unlink returned Ok
          failed: Vec<(PathBuf, io::Error)>,           // unlink returned Err
          fsync_failed: bool,                          // the parent fsync after the
      },                                               // unlinks failed: every path in
  }                                                    // `removed` is uncertain (§3.2)
  fn housekeeping_commit(&mut self, outcome: ReclaimOutcome)
      -> Result<HousekeepingProgress, ReclaimError>   // the ledger half again, under the
                                                       // guard: removed entries leave the
                                                       // ledger and its accounting unless
                                                       // fsync_failed, in which case the
                                                       // whole removed set stays reclaiming
                                                       // as uncertain, re-verified next pass;
                                                       // failed ones return to eligible
                                                       // (record failed) or stay reclaiming
                                                       // (unlink failed), per §3.2
  fn reclaim_state(&self) -> ReclaimState                  // §3.5's export surface
  enum RotationKind { Discretionary, Owed }                // §3.3: the cap applies to
                                                           // Discretionary only
  fn rotate(&mut self, kind: RotationKind) -> Result<(), ReceiveError>
                                                           // §3.3's retried rotation,
                                                           // callable without an append:
                                                           // discharges a pending directory
                                                           // fsync first, no-op when the
                                                           // current segment holds no frame;
                                                           // Wal::rotate is private today.
                                                           // RFC 0053 uses this definition, and
                                                           // its segment cap defers only a
                                                           // Discretionary rotation — an Owed
                                                           // one (a seal, or a recovery
                                                           // requirement) always proceeds.
  

  The own-frame mark needs plumbing, stated so it cannot be skipped. Journal::append_batch discards the WalOffset Wal::append returns today, and CommitOutcome exposes only the group-sync result. So append_batch returns the offset, the coordinator records each waiter’s own offset by sequence at append time, CommitOutcome carries it beside the durable EOF, and every test double returns a monotonically increasing offset per append. The call site uses that own offset for everything it compares with last_durable — the segment-change detection that fires the rotation hook included — so a flush spanning an old-segment turn and a new-segment turn fires the hook once, on the new-segment turn, with the old segment’s mark, rather than on both. An implementation that kept storing the EOF fails RFC0052.14’s two-turn flush, which is what that criterion is for.

  RetainFloor exists because Option<WalOffset> cannot carry what §3.2 requires: None means no snapshot consumer exists and the checkpoint alone governs, which is safe to reclaim; Min is the complete case; Pinned says the minimum is being held down by tenants without a valid snapshot, at their oldest surviving frames, which is correct but must be visible. An Option collapses the last two, and a signature that hides the pinned case is the wrong signature.

  The floor the export reports is the floor the WAL derived on its last pass. The receiver passes SnapshotHorizons — what it knows — and the WAL combines them with its own ledger (§3.2) into RetainFloor, returns it in HousekeepingProgress and keeps it for reclaim_state(), with its lag (§3.5) and its Pinned count. Between passes that is by definition the floor governing retention, so the export is never stale and needs no hidden coupling; before the first pass it reports Unknown, which maintain treats as “reclaim nothing” and the export shows as its own state — not None, which would claim no consumer exists.

  The cap’s value is WAL configuration — WalConfig::max_unlinks_per_pass, proposed default 128, validated at Wal::open to be at least rotation_retry_attempts (RFC0052.4’s one-pass debris clearance depends on it), refusing to open with OpenError::InvalidConfig like the other §6.9 tunables, because only the WAL knows both numbers — and the coordinator passes it to housekeeping_prepare explicitly rather than the trait reading it, so a test can drive any cap through a double and the trait stays free of configuration. HousekeepingProgress reports whether the pass hit the cap, so the caller can tell “backlog drained” from “more to do” without re-deriving it. Without both, RFC0052.12’s bounded-stall contract has no way to be implemented.

  The concrete impl maps CheckpointError and HousekeepingError into ReclaimError; a Box<dyn Journal> cannot infer an associated error type, which is why one enum rather than two. ReclaimState is a plain snapshot struct carrying what §3.5 exports — the existing unflushed_bytes, segment_count and best-effort disk_bytes (a diagnostic, and the only path the trait object has to it), all unreclaimed bytes and the age of the oldest unreclaimed frame (the bytes being what RFC 0053’s bound is taken on, and not a below-checkpoint figure), the retain floor with its lag and Pinned state, and the rotation-failure state — so the server reads it without reaching past the trait.

  disk_bytes is not reused for the admission measurement. WalMetrics documents it as best-effort: its directory walk skips unreadable entries and reports no error when the walk itself fails, so a silent undercount would let admission sail past the bound — the one place a best-effort number is unacceptable. The unreclaimed-byte figure is maintained incrementally from appends and unlinks, as the WAL already does for unflushed_bytes, and disk_bytes stays what it is: a diagnostic. The figure is seeded after recovery, not at Wal::open: open runs before recovery::recover replays and heals the newest segment, so a seed taken there would count torn bytes. Recovery therefore ends — after every successful replay, and after the heal when there was a torn tail to heal, since a clean replay never enters that path — by calling Wal::rebuild_ledger(), a concrete method since recovery holds the WAL before it is boxed. A size figure is not enough for it: the ledger §3.2 relies on needs, per surviving segment, which tenants have frames in it and each one’s first and last offset, and the partial list needs the directory’s .wal.partial entries. So rebuild_ledger() is a scan, not a stat: it walks every surviving segment oldest-first, validates each frame prefix (length, checksum) exactly as replay does, decodes the tenant from the TenantOtlpBatch prefix append_batch framed, and records the per-tenant offsets and the frame bytes into the ledger; the unreclaimed-byte figure is the sum of the validated frames’ lengths — never file size less header, which would count a torn tail — and the current segment’s own frame bytes are seeded from the healed newest segment in the same pass. Replay already reads every frame, so an implementation may build the ledger as a by-product of the replay pass rather than a second scan; the contract is the ledger’s content, not the number of passes. The coordinator is constructed after recovery (the ordering below), so no append can precede the seed, and a restart mid-outage resumes from the true backlog rather than from zero. unflushed_bytes stays its own method: the group-commit coordinator reads it per batch and must not allocate a snapshot struct on that path.

  On the trait rather than via a downcast, because RFC0052.1 and RFC0052.2 need test doubles that can observe reclamation.

• Recovery gains the Parquet-side gate §3.1 requires. recovery::DriverSink owns only the miner today and merely records parquet_horizon — and MinerCluster::ingest already emits every mined record to the sink serve wired in before recovery, so a second sink handle in the driver would emit replayed rows twice. Recovery therefore drives the miner through a capture path (ingest_mined, or the configured sink disconnected for the replay) so that replay emits nothing implicitly, feeds the miner only above that tenant’s snapshot horizon S, as now, and forwards the captured records to the record sink only above max(X, S) per tenant, and captures the miner’s audit events the same way — through a replay capture sink the driver installs, which receives both mined records and audit events tagged with the offset of the frame being replayed, since AuditEvent carries no offset and ingest_mined diverts only records — forwarding events only for frames above X (the barrier drains the audit sink before it stamps, so X is the audit horizon): events for (S, X] are suppressed and counted, events above X forwarded, and neither duplicated nor lost. Regeneration by the miner is the only source of replayed events: stored AuditEvent frames are not replayed — nothing writes them today (encode_audit_event is unimplemented!(), RFC 0008 §9’s deferral) and recovery keeps ignoring the kind — and an encoder that lands later must either keep replay on regeneration or amend this leg, never feed both, or an event would be injected twice; RFC0052.10 gains the audit leg — max, because the snapshot is written before the checkpoint is persisted, so a successful snapshot followed by a failed checkpoint write leaves S > X with (X, S] already in Parquet. Frames below that gate are suppressed on the Parquet side and counted, so a restart never republishes a row, and RFC0052.10’s no-duplicate leg — including the S > X ordering — is what proves the gate exists rather than assumes it.

• The barrier reaches them through the coordinator, which already owns the journal mutex, rather than taking a second handle to the same WAL. A second handle would put two owners on a single-writer resource, which is the one thing RFC 0008 §3.1 forbids. Concretely, CommitCoordinator gains maintain(&self, horizons: &SnapshotHorizons, cap: usize) -> Result<HousekeepingProgress, ReclaimError> — the documented housekeeping contract, no separate report type — and checkpoint(&self, mark: WalOffset) -> Result<(), ReclaimError>; serve reaches it from the timer through SharedPipeline and runs it on the blocking pool because it does file I/O. maintain is where §3.2’s two halves become a protocol, because the coordinator’s journal is Mutex<Box<dyn Journal>> and a single Journal call would keep every unlink and fsync inside that guard, in front of every concurrent append: maintain locks the journal for housekeeping_prepare and releases it; runs the file half itself on the plan’s owned paths — merge and write the inactive RECLAIM slot in place, fsync, then the unlinks, then the parent fsync — with no guard held; and locks again for housekeeping_commit with the outcome. Between the two the WAL keeps serving appends and rotations, which never touch an entry marked reclaiming. One plan is outstanding at a time by construction (one housekeeping task), and a plan that is never committed — the task panicked between the halves — strands nothing: the next housekeeping_prepare re-plans every entry still marked reclaiming ahead of anything newly eligible, the record merge is idempotent and monotonic, and an unlink that finds the file already gone counts as removed. A crash between the halves reconciles at open the same way, with no special case: rebuild_ledger() rebuilds the ledger from the segments that survive, so a segment unlinked before the crash is simply absent and one not yet unlinked is present and still eligible under the horizons the durable RECLAIM record already carries, since the record was written before the first unlink; the open-time directory fsync makes the surviving set definite before the walk.

• Option<WalOffset> skips, and the post-recovery call is an explicit exception. None means no high-water mark is known, and there is then nothing to declare reclaimable. None is not the same as “post-recovery”, which the earlier draft conflated. The post-recovery call passes a mark derived from replay, and not report.max_delivered on its own: a frame can be delivered by replay whose group sync never completed — its bytes survived the crash, its batch was never acknowledged — and §3.1’s rule is that such a frame is never a mark. The seed is therefore the highest offset that is both delivered and covered by a sync known to have succeeded — and the only such offset that survives a crash is the CHECKPOINT mark. last_durable is in-memory state the running process rebuilds, persisted nowhere, so it cannot seed anything after a restart; an earlier draft said “the checkpoint, or the last_durable the sidecar carries”, and the second half of that was wrong. So the seed is the checkpoint mark when one exists and None otherwise — a node whose first barrier never ran — with the first successful turn establishing it, which costs one barrier and never stamps across an unacknowledged frame. The call runs before the commit coordinator exists, so there is no journal owner to checkpoint through. That call therefore deliberately advances no checkpoint; the first timer pass after the coordinator is built does it instead, from the same mark or a later one. The post-recovery snapshot write stamps each tenant at its own folded horizon (§3.1), read from MinerCluster::folded_horizon and written by recovery::write_folded_snapshots, the one writer the shutdown point shares: restoring a snapshot sets the tenant’s horizon to the one it carries, and replay advances it to each frame it feeds. So when replay delivers nothing — the normal shape once housekeeping has reclaimed every closed frame and the node restarts idle — each tenant keeps its restored horizon rather than having None written over it, which would discard the snapshots on the next restart and force a full replay every time; and a replay that delivered frames for tenant A only records A’s own last frame and keeps B’s restored horizon, which no global max_delivered could express. The snapshot ledger is seeded from what that write installed, and fail-closed: a skipped or partly failed write falls back to the horizons recovery restored, never to the higher ones it attempted, so the ledger is never above what is on disk. Before any of it — inside recovery, after the ledger rebuild — a pre-RFC root runs the §3.2 legacy stale-gap check against the marks its version-1 artefacts recorded, since the post-recovery write replaces those artefacts and would destroy the evidence. §3.1’s “behind the barrier and nowhere else” holds — this is a caller that has the mark but not yet the owner, not a second checkpoint site.

• Errors are fail-closed and never swallowed. A failed checkpoint logs and leaves the in-memory checkpoint unadvanced, which is already Wal::checkpoint’s own behaviour; the WAL then reclaims nothing past the previous mark, which stays usable — RFC0052.1 requires exactly that, not that every segment is kept. A failed housekeeping logs and the next pass retries, since nothing was unlinked past the bound. Neither failure fails the barrier: the data is in object storage either way, and refusing to ack over a reclamation error would turn a disk-space problem into an availability one.

One consequence worth naming. Housekeeping takes the single-writer position for its ledger half, so every append stalls for that half’s duration — which is why §3.2 and §3.7 put the whole file half (the RECLAIM write, the unlinks and the parent fsync) outside the journal mutex, between housekeeping_prepare and housekeeping_commit, and why the ledger half does no directory walk at all. Running the file half on the blocking pool while still holding the lock would bound nothing: the stall is caused by holding the journal mutex, not by occupying a runtime worker. What remains under the lock is the capped ledger work RFC0052.12 bounds.

The pass is therefore capped on the unlinks, and the eligibility evaluation costs no I/O at all. The current path enumerates and sorts every *.wal and reads each candidate’s header before unlinking any, so a 1,113-segment backlog did 1,113 header reads under the mutex however few files it then removed — which is why an earlier draft capped the inspected candidates too. That cap is withdrawn, because it conflicts with tenant-aware eligibility: the oldest segment may be pinned while a later one is eligible, and a pass that inspected only the oldest few would stop at the pinned one every tick and never reach the eligible one. It is also no longer needed: everything a header read supplied — the segment’s highest offset, each tenant’s last offset in it, its frame bytes — lives in the per-segment ledger the WAL maintains in memory (§3.2). An O(n) walk of that ledger under the writer lock would still not be a bound — n is exactly what a pinned tenant or a store outage leaves unbounded — so eligibility is indexed incrementally rather than evaluated per pass: the WAL keeps, per segment, the set of tenants whose horizon is still below their last offset in it; applying new horizons (the SnapshotHorizons input, under the mutex) shrinks those sets for the segments those tenants span — lazily and capped: at most the cap’s worth of segments per pass, resumed from a per-tenant cursor on the next, so a tenant whose snapshot catches up after the incident’s 1,113-segment outage cannot hold the writer position for the whole backlog — and empty-set segments sit in an ordered structure keyed by highest offset. The cursor is defined once, here. Per tenant it holds the highest segment offset up to which that tenant’s horizon has been applied; horizons are monotonic per tenant and a tenant’s last offsets rise with the segments, so application is a prefix walk oldest-first, and a higher horizon only extends the walk’s target — the cursor never moves backwards and is never reset by a checkpoint advance, which touches no set. Each pass spends its application budget round-robin across the tenants whose cursor is behind their horizon, so one tenant cannot starve another’s. The only rebuild is at open, when rebuild_ledger() rebuilds the ledger and every cursor starts at the oldest surviving segment; the first pass after startup applies from there. The eligible head needs no cursor of its own: it is re-evaluated from the ordered structure on every pass. There is no promotion pass: a checkpoint advance stores the new mark, O(1) under the mutex, and a segment is eligible when it is empty-set and its key is at or below the stored mark — a predicate the pass evaluates lazily at the head of the ordered structure, popping while the head’s key is at or below the checkpoint and at most max_unlinks times. An earlier draft promoted every newly covered empty-set segment into an eligible queue on the checkpoint advance, in one ordered pass; that pass is O(backlog) under the journal mutex — on the incident’s 1,113 segments it stalls every append before the capped unlink even starts — and contradicts the bound this section exists to give, so it is withdrawn. Nothing is stranded without it: a segment whose set empties above the checkpoint waits in the ordered structure and is at its head once the mark passes it, and a segment the mark passed while its set was non-empty is found once a horizon application empties it. The pass therefore does at most the cap’s worth of horizon application plus at most the cap’s worth of pops — O(cap) under the writer lock, with no name listing, since segments are known from the ledger — and HousekeepingProgress reports capped from either half hitting its budget, plus the two remaining-work figures horizon_remaining and unlink_remaining (§3.7 defines their aggregation), both exported (§3.5). RFC0052.12 is worded to that guarantee. The next tick continues where this one left off — the per-tenant cursor is the resume point for horizon application, and the eligible head is re-evaluated from the ordered structure, oldest first, so it needs none. That matters most on the first pass of a node that has never reclaimed — the incident node held 1,113 segments, and an uncapped pass would have stalled ingest for as long as 1,113 unlinks take. In steady state, after §3.2 is running, a pass has a handful of segments to consider and the cap never binds.

3.8 Tunables classification

RFC 0008 §6.9 classifies every operator-visible WAL knob as exactly one of Tunable or Invariant, with a startup-validated range and the CLAUDE.md §3 invariant it lives inside (RFC 0004 §3.1’s schema). This RFC adds knobs and gives two existing ones their first caller, so it carries its own rows in the same schema. Names are the WalConfig fields; §6.9’s wal_-prefixed spellings are the same knobs on the operator surface. All are process-global, as §6.9 requires — the WAL is one workspace-wide log — and every range below is validated in Wal::open, which refuses to open with OpenError::InvalidConfig rather than clamping, exactly as the existing tunables do.

max_unlinks_per_pass is the one row whose validation reads another knob, which is why it lives in the WAL rather than in the caller: only Wal::open sees both numbers. The four Invariant rows are the persisted shapes this RFC introduces or bumps; each carries the ! marker for the implementing PR under the pre-production layout policy, and none is operator-settable.

4. Alternatives considered

Checkpoint on its own timer, independent of the publication barrier. Rejected: it would need to re-derive “everything at or below this offset is durable in the store”, which is precisely what the barrier computes under the miner lock. A second derivation is a second thing to get wrong, and the two could disagree.

Checkpoint at every successful cadence flush rather than at the barrier. Rejected: a cadence flush is per-partition and best-effort, and does not establish the whole-sink condition. It would advance the checkpoint past partitions that had not drained.

Leave the quiesce permanent and add an operator verb to clear it. This is what the current comments promise, so it deserves a real answer: it is honest and simple, and it puts a human in the loop before writes resume after a durability fault. Rejected as the primary mechanism because the fault it most often fires on is transient and self-correcting, and requiring a human for a condition that has already cleared is how the incident produced eight hours of downtime. A bounded retry that gives up into a distinctly-reported terminal state keeps the human in the loop for faults that are actually persistent. An explicit verb remains worth adding later for the terminal state.

Tolerate a torn header in Wal::open instead of unlinking the orphan. Rejected: it weakens the corruption detection RFC0008.5 depends on in order to avoid one remove_file, and a zero-frame segment with an unreadable header is indistinguishable from a real corruption at open time.

Retry rotation from a background task. Rejected: the WAL is single-writer by design (RFC 0008 §3.1) and owns no task. A background retry would need to take the writer position away from the append path, which is a far larger change than retrying where the need arises.

The alternatives to backpressure and to requeue-on-unwind — a hard sink ceiling, dropping on unwind — moved with those designs to RFC 0053 §4. The sink-ceiling finding there (memory can OOM before any disk bound fires) is a prerequisite of RFC 0053, not of this RFC: nothing here bounds memory, and nothing here claims to.

5. Acceptance criteria

Scenario RFC0052.1 — The checkpoint advances only behind a proven publication barrier

• Given a WAL with acknowledged frames and a record sink whose store is healthy
• When the publication barrier completes with both sinks fully drained
• Then the journal’s checkpoint is advanced to the barrier’s high-water mark
• And when either sink retains anything, no checkpoint is attempted and last_checkpoint() is unchanged
• And when the high-water mark is None, no checkpoint is attempted
• And while the cadence_failed latch holds an epoch at or below the cut’s, the barrier neither checkpoints nor snapshots, however many timer passes run, until a restart
• And an age-sweep publish registered before the barrier began, which panics while the barrier waits in quiesce_publishes, leaves the checkpoint and every snapshot unchanged: the latch set after the barrier’s first check is observed at its recheck before stamping, and the publish’s outcome is reported failed independently of it
• And an encode worker panicking mid-batch, followed by a barrier, leaves the checkpoint and every snapshot unchanged, and the batch’s unemitted records are replayed on restart
• And the same holds when the barrier is started concurrently with the worker’s panic, under every schedule: a barrier that observes the pool’s pending count at zero has observed the latch, because the unwinding guard stores it before the decrement that settles the count
• And a panic in the barrier tick itself — outside any batch guard — lowers failed_epoch to that tick’s epoch, invalidates the pending slot, and the barrier task takes the next tick; a panic in a housekeeping tick counts with error.type = cadence_panic, leaves the checkpoint untouched, and the next pass re-plans the uncommitted plan; a JoinError from either task at shutdown is logged and read as a failed cut
• And a batch queued before cut E’s capture and dequeued after it carries epoch E — assigned in submit, not at dequeue — so a panic in it fails cut E, not only later ones
• And a partition detached by that batch’s first record, with the cut captured between the detach and the enqueue, is covered: the publish guard was created in submit before the exclusion was released, so the cut waits for it rather than stamping past it, and a batch that detaches nothing releases its guard unused
• And a batch that detaches several partitions settles its guard only when the last of them completes, in any completion order, so a cut started after the first completes still waits for the rest
• And a publisher panic with batches still queued behind the failing one latches only the failing batch’s epoch, parks every queued batch in the buffers with its guard released, and a quiesce_publishes started during the panic returns; the next enqueue finds a respawned publisher
• And a worker whose send fails on a closed channel parks that batch in the buffers under the sink lock before releasing its guard, so the records are covered by the next drain rather than silently settled, and the publisher is respawned
• And a detached partition whose audit_watermark is covered by another writer’s in-flight audit write is not published until that write is durable: an empty audit buffer is not read as a durable prefix, and when that write fails the dependent partition requeues rather than landing in Parquet ahead of its template events
• And when the checkpoint write fails, the barrier still reports success, last_checkpoint() is unchanged, and the next housekeeping pass reclaims nothing that was not already eligible under the previous mark — Wal::checkpoint leaves that mark intact on failure, so forbidding all reclamation would reject the fail-closed behaviour §3.1 specifies rather than test it

Scenario RFC0052.2 — Segments are reclaimed, and never past the minimum tenant snapshot floor

• Given a checkpoint advanced past several closed segments and two tenants whose snapshot horizons differ, the lagging one below that checkpoint
• When housekeeping runs
• Then only segments at or below the checkpoint (inclusive) whose every tenant’s last frame in them is at or below that tenant’s horizon (inclusive) are unlinked, the current append segment survives, and the WAL’s segment count falls
• And a segment whose only frame for an idle tenant is exactly that tenant’s horizon frame is unlinked — the horizon is durably installed by construction, so its segment is not needed — and a restart afterwards restores the tenant from its snapshot with no stale-gap report, the RECLAIM entry at or above S explaining the absent segment
• And a segment holding a frame above a tenant’s horizon is retained until the tenant’s horizon reaches it
• And a frame above the lagging tenant’s horizon is still present after the pass, so a restart re-mines it rather than losing that tenant’s miner state
• And when a tenant with WAL data has no valid snapshot, the pass retains exactly that tenant’s segments (RFC0052.13’s pin) and still reclaims a later segment holding only other tenants’ covered frames, while the temp sweep still runs

Scenario RFC0052.3 — Sustained ingest does not grow the WAL without bound

• Given a node ingesting continuously at a rate its publish and reclamation path can sustain (the soak is capacity-balanced and pinned as such), with a healthy store and every tenant with WAL data holding a valid and advancing snapshot (a complete floor), run long enough to roll many segments
• When the reclamation cadence has had time to act
• Then the WAL’s on-disk byte total and segment count are bounded rather than monotonically increasing — the #793 signature (1,113 segments, nothing ever unlinked) cannot reproduce
• And while the floor is Pinned the pinning tenants’ frames are retained by design and the state is visible (RFC0052.13); a healthy store alone does not bound the WAL, a complete floor does
• And an offered rate above that capacity grows the WAL by construction until RFC 0053’s admission bound exists; this criterion claims no bound there
• And after the last append, with no further traffic, one barrier_secs plus one housekeeping_secs later the checkpoint has advanced and every eligible closed segment is reclaimed — the append-independent path, which continuous traffic alone cannot prove — and within segment_age_secs + barrier_secs + housekeeping_secs the last segment has rotated on a barrier tick and been reclaimed by the following pass

Scenario RFC0052.4 — A transient rotation failure recovers without a restart

• Given a WAL whose rotation fails once with a transient I/O error — triggered by an append, or by the barrier task’s idle rotation
• When the condition clears and, for a failure before the rename, a later append arrives — or, for the post-rename directory fsync, the next sync runs
• Then the pre-rename case re-enters rotate, which succeeds, and the append is accepted and acked; the post-rename case discharges the pending directory fsync in sync without re-entering rotate, and the batches behind it are acked
• And when the failure was before the rename, no file a subsequent Wal::open would select as a segment remains from the failed attempt — asserted by opening the WAL again, not by inspecting the directory, since the temporary name is an implementation detail
• And when the failure was the post-rename directory fsync, the installed segment is selected by a subsequent open — it is complete and valid — and the first sync after open discharges its pending directory fsync
• And the temporary files left by the failed attempts are gone after successive capped housekeeping passes — WalConfig::max_unlinks_per_pass is validated at Wal::open to be at least rotation_retry_attempts (open refuses otherwise), so one rotation’s debris clears in one pass — and a persistently retrying node cannot fill its disk with retry debris

Scenario RFC0052.5 — A persistent rotation failure gives up distinguishably, and never acks

• Given a WAL whose rotation fails on every attempt — in rotate itself, or in the directory-fsync discharge that sync retries
• When appends continue past the bounded retry count
• Then every append is refused, no batch is acknowledged, and the refusal is reported as the terminal state rather than as a transient one
• And the first underlying I/O error is still recoverable from the reported state, not replaced by a generic “quiesced” message
• And Wal::open on the resulting directory succeeds rather than reporting corruption, including when the cleanup of the last attempt’s temporary file never completed

Scenario RFC0052.11 — A node already wedged before this RFC halts actionably at open

• Given a WAL directory in the state today’s rotation failure leaves: a newest *.wal whose header bytes are partial, written before this RFC landed
• When the WAL is opened
• Then it still reports OpenError::Corrupt rather than unlinking anything — §3.3 withdrew the shape-based heuristic, because an unreadable newest segment is indistinguishable from real corruption
• And the error names the file, describes the observable shape and names a failed rotation as one possible cause without asserting it, so an operator can decide deliberately
• And a node whose rotations all happened under this RFC cannot reach that state at all, so the population is finite and shrinking

Scenario RFC0052.12 — A reclamation pass bounds its per-file work

• Given a backlog far larger than max_unlinks_per_pass (the incident’s 1,113 segments is the shape)
• When a housekeeping pass runs
• Then it unlinks at most the cap and returns, having read no segment header and listed no directory — eligibility comes from the incremental index, and the pass pops at most the cap from the eligible queue — so an append taken concurrently waits for O(cap) ledger work under the writer lock whatever the backlog, and never for the RECLAIM write, an unlink or an fsync, which run between housekeeping_prepare and housekeeping_commit with the journal guard released — asserted by an append that completes while the file half is held at a fault-injection point; applying changed horizons is itself capped — at most the cap’s worth of segments per pass, resumed from a per-tenant cursor next pass, so a tenant catching up after a long outage cannot hold the writer position for the whole backlog — and an unchanged pinned backlog costs none
• And a checkpoint advance is O(1) under the mutex — it promotes nothing eagerly, whatever the backlog — and a segment whose set emptied above the old checkpoint is reclaimed by the first pass after the mark passes it, so it is not stranded when horizons stop changing
• And stale partials left by a previous process are removed from the list seeded at recovery, without a directory listing on the pass
• And a pinned oldest segment does not shadow a later eligible one: the pass reclaims the eligible segment on its first tick
• And successive passes drain the backlog to the same end state an uncapped pass would reach
• And a backlog of stale temporary files is also bounded by the same cap, so temp sweeping cannot make a “bounded” pass do unbounded work
• And with a segment backlog larger than the cap and stale partials present, the first pass removes the partials and spends only the remainder of its cap on segments

Scenario RFC0052.16 — The temp sweep touches only files of the reserved partial shape

• Given a WAL root holding a CHECKPOINT.tmp and a RECLAIM (which has no temp of its own — it is rewritten in place, §3.2), a snapshots directory holding a *.snap.tmp, and a stale <uuid>.wal.partial
• When a housekeeping pass runs
• Then only the partial is unlinked: the checkpoint temp, the reclaim record and the snapshot temp survive
• And the unlink is followed by a parent-directory fsync, so a crash cannot resurrect it

Scenario RFC0052.15 — Only the terminal rotation state is reported server-terminal, client-retryable

• Given §3.3’s bounded retry, a rotation failure that is still within the budget, and one that has exhausted it
• When each is reported on both transports
• Then all of them carry 503 / UNAVAILABLE — RFC0018.3 still holds, and a non-retryable code would tell the client to drop an unacked batch
• And a failure still within the retry budget is classified transient and its message says it is retrying, because §3.3 means a later append genuinely can succeed; no retry hint is carried, since the server schedules no retry of its own
• And only the terminal state is classified server-terminal, client-retryable — RFC 0018 §3.2’s third class — with the message naming that state and still no retry hint, so a conforming client keeps the batch and backs off exponentially
• And an ordinary append or fsync I/O failure stays transient, so the reclassification is narrow rather than a blanket change to RFC 0018 §3.2’s transient class

#794 originally classified every rotation failure as wedged, which was correct for today’s permanent latch and wrong the moment §3.3 makes rotation retryable. That half of #794 was split out and is held on hold/794-wedged-classification; it lands with this RFC, rewritten to the terminal-only rule above. The merged half of #794 is the Status body conformance alone and stays correct under both.

Scenario RFC0052.14 — The timer cannot stamp across a concurrent submit

• Given the reclamation timer firing while ingest submits continuously
• When the timer runs its sequence
• Then no checkpoint is advanced past a frame whose encode had not finished its sink emit, under any interleaving
• And the mark used is the one read after the quiesce under the same exclusion, not one read before either
• And that mark is a turn’s own frame offset, never the sync’s reported EOF, and nothing replay delivered is ever a mark: a restarted node has no mark until its first acknowledged turn establishes one, so a replayed frame whose group sync failed before the crash is never checkpointed across (a replayed seed, where a caller still supplies one, stays distinct from a mark; whether that seed survives is #883) — so a later frame made durable by the same flush but not yet mined (nor acknowledged) is never covered — asserted by a flush whose sync covers two turns and a barrier between them
• And a rotation capture that would take the pending cut past the sink’s ceiling parks every batch it drained and advances neither the mark, the snapshots nor the epoch, so the checkpoint that pending cut eventually stamps covers only frames its own batches held, and the parked partitions are covered by the next cut
• And an idle rotation on the barrier tick rotates before the cut, under the exclusion: the mark is the last acknowledged turn’s frame offset in the closed segment, never the rotation boundary, a frame appended after the release lands in the new segment above the mark, and a frame written but never acknowledged is never a mark
• And a pre-cut batch whose first record detached a partition mid-batch has its remaining records in the cut, under any interleaving: the quiesce waits for the batch’s encode phase, not for a worker to register a publish, and the detached partition’s PUT is not waited on
• And with several partitions detached from one pre-cut batch and completing in any order, the cut covers every record: the batch’s shared completion holds the in-flight count until the last of them finishes
• And a rotation-fired cut performs no store I/O inside the ingest turn: the turn hands the captured cut to the barrier task, and an append admitted after the turn is neither in that cut’s checkpoint nor in its snapshot
• And cuts are strictly ordered: with cut A’s flush held at a fault-injection point and cut B captured behind it, B neither installs nor checkpoints before A’s outcome; when A then fails, B is invalidated — no snapshot of B’s is installed, no mark of B’s is stamped, A’s and B’s records are all in the buffers — and the re-captured cut covers them all; when A succeeds instead, B runs unchanged
• And a failed publish refuses only cuts captured before its requeue: the first cut captured after the requeue stamps, so one transient failure cannot refuse every later cut
• And an age-sweep publish registered after the cut — between the barrier’s quiesce_publishes and its checkpoint — that fails transiently or panics is neither covered by that checkpoint nor lost: every frame it holds is above the cut’s mark, a transient failure requeues it, and a panic — carrying the next epoch — fails no cut at or below the current one, which proceeds, and fails every later barrier until restart; the barrier does not re-take the exclusion around its store I/O to reach that

Scenario RFC0052.13 — A tenant without a snapshot pins the floor, and is never read as unbounded

• Given a snapshot consumer exists and one tenant with WAL data has no valid snapshot
• When housekeeping runs
• Then every frame of that tenant survives, segments below its oldest surviving frame that hold only other tenants’ covered frames are reclaimed, and the floor is reported as Pinned — distinguishable in the API from the no-consumer case, which reclaims by checkpoint alone
• And once a valid snapshot for that tenant is written, the pin lifts and the next pass reclaims what it had held
• And a tenant leaves the ledger only when its last surviving segment is unlinked, so tenant churn cannot leave a permanent pin
• And a segment holding exactly one frame for the pinning tenant survives the pass — a pinned tenant has no horizon, so equality can never unlink its frame — while a later segment holding none of its frames is reclaimed
• And a snapshot listed at startup is used as a horizon only after the snapshots root and its parent have been fsynced in this process; a failed startup fsync fails startup rather than discarding snapshots whose frames reclamation may already have removed, so a horizon whose directory entry may not be durable never governs reclamation and no state that only a snapshot could rebuild is thrown away
• And the state is exported (§3.5), so an operator seeing a WAL that will not shrink can tell it is a pinning tenant rather than an unexplained stall

Scenario RFC0052.6 — moved to RFC 0053 as RFC0053.1 (backpressure). The number is retained so review-history references resolve; it carries no obligation in this RFC.

Scenario RFC0052.7 — The WAL’s state is exported

• Given a running node
• When metrics are collected
• Then the WAL’s unflushed bytes, on-disk bytes, segment count, all unreclaimed bytes and the age of the oldest unreclaimed frame (the bytes being what RFC 0053’s bound is taken on, including the post-checkpoint tail), the retain floor with its lag and Pinned state, the cadence_failed latch, the rotation-failure state, and the housekeeping backlog (horizon application remaining and unlinks remaining, in segments) are all present in the exported stream under registry names
• And a run whose checkpoint never advances still reports growing unreclaimed bytes — exporting the below-checkpoint figure instead would read as flat during exactly the outage it exists to show
• And entering the terminal rotation state emits exactly one log event, named from the registry; leaving it is a restart today, which a fresh WAL cannot observe, so no leave event exists until §7’s operator verb does, and that verb is where it would be emitted — the refusing state’s events are RFC0053.5’s

Scenario RFC0052.8 — moved to RFC 0053 as RFC0053.2 (unwind keeps the records). Number retained; no obligation here.

Scenario RFC0052.9 — moved to RFC 0053 as RFC0053.3 (the cadence survives a panic). Number retained; no obligation here.

Scenario RFC0052.17 — The reclaim record is the only startup witness, and it is fail-closed

• Given a WAL root housekeeping has reclaimed from under per-tenant horizons, so RECLAIM holds an entry per reclaimed tenant
• When the node restarts with one tenant’s snapshot undecodable
• Then recovery halts naming that tenant when the record holds an entry for it and the tenant has a surviving frame, and proceeds with that tenant pinned at its oldest surviving frame when the record holds none
• And a root with no record and no checkpoint at all — the pre-RFC layout on a node that never checkpointed — opens, gains an empty record durably before its first housekeeping pass, and pins rather than halts; a root with a version-1 CHECKPOINT instead takes the migration path above, where the record is created by the upgrade and not at open
• And a record failing its checksum or version byte fails open as OpenError::Corrupt naming the file, and is never read as missing
• And a pass reclaiming under a higher horizon for one tenant leaves every other entry unchanged, and no pass ever lowers an entry
• And a crash injected between the record’s write and the first unlink leaves a record whose entries every restorable snapshot satisfies, so the restart proceeds; a crash injected inside the inactive slot’s write, or between that write and its fsync, leaves the previous slot live — the torn slot fails its CRC or carries the lower generation, and the reader takes the other one — so the next pass re-plans from the record the previous pass left
• And a root that has reclaimed and whose RECLAIM is then deleted fails open naming the missing record — its CHECKPOINT is version 2, so the root is post-RFC — rather than recreating an empty record and pinning
• And a replay that yields a tenant longer than 128 bytes — a frame written before the codec was amended — fails open naming the frame offset and the length, rather than truncating the key or dropping the frame
• And a root holding segments and neither sidecar fails open naming both when any segment header carries version 2, while a root whose segments are all version 1 — every live pre-RFC root — opens on the legacy branch, and an empty directory with neither sidecar opens as a fresh root
• And the same holds for a NoConsumer root, which has no snapshot artefacts to witness with: its version-2 segments are the witness, so losing both sidecars fails closed rather than recreating an empty record over its reclaimed-through proof
• And a legacy root that rotates before its first checkpoint stays openable: the rotation writes and fsyncs the record before creating the version-2 segment, so no restart ever finds a version-2 segment beside no record — asserted by injecting a crash between the two steps
• And a configured max_tenants or max_unlinks_per_pass above the capacity stored in the header rebuilds the file at the larger geometry through RECLAIM.new — write, fsync, rename, parent fsync — so a crash at any point leaves either the old file or the new one complete and the entries survive; a configured value at or below the stored capacity opens without rewriting
• And a full volume at Wal::open fails the allocation as OpenError::Io naming the file, so the node does not start — not as a rotation state, which no running process exists to report — and it starts normally once space is freed
• And every pass writes its slot without allocating, whatever the tenant set does, so a node whose volume filled after open still reclaims its way out; slot_len never changes for the life of the file
• And a node’s own first start is never that state: the empty record is durable before the initial segment is created, so a restart immediately after a fresh Wal::open — with or without a crash between the two — finds the record and opens normally
• And a root holding a version-1 CHECKPOINT (an RFC0008.7 fixture) and no RECLAIM opens as pre-RFC without creating a record, and its first checkpoint rewrites the sidecar at version 2 and creates the record armed on the same path
• And that first checkpoint upgrades the sidecar even when its mark equals the one on disk — the idle-node case — while an equal mark on an already-version-2 sidecar still takes the no-write path
• And on a legacy root a tenant whose snapshot is missing and whose oldest surviving frame is above its last recorded horizon fails closed naming the tenant, so a root that reclaimed under a version-1 checkpoint is caught rather than pinned
• And a restart in that migration window — still a version-1 CHECKPOINT, still no record — takes the legacy branch again rather than any fail-closed row
• And a housekeeping pass that fires in that window plans no segment, writes no record and is counted as a skipped pass with its reason, while still sweeping stale .wal.partial files, so rotation debris from before the first checkpoint does not survive the window; the pass after the first checkpoint reclaims segments normally
• And a root whose RECLAIM carries checkpoint_seen — written after its first version-2 checkpoint succeeded, entries or none — and whose CHECKPOINT is missing fails open naming both files
• And a root whose RECLAIM carries checkpoint_armed but not checkpoint_seen with CHECKPOINT absent splits on whether segments exist: with no segments — a crash between arming and the checkpoint’s rename on a root that never held one — it opens as a fresh root with an empty record and is re-armed by the next attempt, as does a RECLAIM with neither flag beside a missing CHECKPOINT; with segments present it is a legacy root mid migration and the record is retained with its arming and its mode, the root opening on the legacy branch with the upgrade retried by the next checkpoint
• (The two PUBLISHED legs — slot ids surviving a PUBLISHED-only write, and the published_seeded_* flags resolving at open — moved to RFC 0055 as RFC0055.5, since RFC 0055 owns the PUBLISHED writer and format. No obligation here.)
• And the same record beside a present version-2 CHECKPOINT — the crash after the rename and before the record’s next write — opens normally and is promoted to seen durably at open, never read as a fault
• And an armed record beside a still-version-1 CHECKPOINT — the upgrade write failing between the arming and the rename — opens on the legacy branch, reclaims nothing until the next checkpoint retries the upgrade, and that retry succeeds against the arming already on disk; asserted by restarting the node in that state
• And a WAL used with no miner state reclaims under SnapshotHorizons::NoConsumer and restarts without a snapshot without halting: its entries carry that mode and are checkpoint-covered
• And a pass whose mode disagrees with the mode recorded in the RECLAIM header is refused as a ReclaimError naming both modes, before anything is planned or unlinked — including on a root that has checkpointed but never reclaimed, where no entry exists to infer from
• And a root whose header carries no mode adopts the first pass’s mode and persists it durably before that pass unlinks anything, so no reclamation is ever governed by an unrecorded mode
• And the ingest receiver never passes NoConsumer: a tenant with no valid snapshot is expressed as Pinned (RFC0052.13) and its frames are retained, so no miner-bearing tenant can be reclaimed past its replay need
• And a segment whose unlink fails after the record was written stays on disk with reclaimed_through behind it: a restart with that tenant’s snapshot undecodable pins the tenant rather than halting
• And a segment whose parent fsync fails after its unlink is an uncertain deletion: reclaimed_through stays behind it and its bytes stay counted; the next pass re-verifies presence and completes or retries; a restart reconciles it — present ⇒ retained and re-planned, absent ⇒ reclaimed_through raised and no halt, as a completed reclamation — under both outcomes of the injected fsync failure
• And a crash between the record write and the commit is reconciled at open the same way: a planned segment still present is retained and re-planned, an absent one raises reclaimed_through as the commit would have, and the reconciled record is durable before the first pass
• And a record write or fsync that fails unlinks nothing, leaves the WAL’s byte and segment accounting unchanged, and the segments that pass popped are reclaimed by a later pass once the write succeeds — they are never lost to the ledger

Scenario RFC0052.10 — No acknowledged record is lost across the whole cycle

• Given a node killed with SIGKILL mid-batch while reclamation and rotation retry are both live
• When it restarts and recovery completes
• Then every acknowledged record is present in Parquet, including those whose segments were candidates for reclamation at the moment of the kill
• And recovery itself republishes nothing at or below max(X, S): a kill that follows a failed snapshot write and an advanced checkpoint replays frames at or below the checkpoint into the miner only, never into the record sink — a client retry that wrote a second equivalent frame (RFC0003.2’s at-least-once contract) is unchanged by this RFC and is not what this leg counts
• And the same holds when the snapshot write succeeded and the checkpoint write then failed, so a tenant restarts with S > X: nothing in (X, S] is republished
• And the audit stream is gated the same way: no template event for a frame at or below X is forwarded again on replay, and every event the miner regenerates for a frame above X is forwarded exactly once, in frame order, through the capture sink, with stored AuditEvent frames ignored as a source; the test mines the same frames from the same snapshot with the same injected clock as a reference and asserts the forwarded set equals the reference’s events for (X, tail]
• And (2026-10-06, coordinating with RFC 0059 §3.4, maintainer-approved) the one exception is an event at or below X of a template replay minted afresh: its id lies above RFC 0059’s template-id high-water, so no published event binds it, and it is forwarded once, in frame order; the test’s reference forwards the (X, tail] events plus those (S, X] events, and audit_events_suppressed counts every other (S, X] event

6. Testing strategy

Per CLAUDE.md §6.2, mapped to the §5 ids.

• Checkpoint policy (RFC0052.1) — unit tests over the barrier with a healthy store and with a store that fails a partition write, asserting last_checkpoint() advances in the first case and is untouched in the second; plus a third leg that fails the checkpoint sidecar write and its directory fsync with a healthy store, asserting the barrier still reports success, the previous mark stays usable and the next pass reclaims only under it — the fail-closed branch a partition-write failure cannot reach. Two unwind legs use seeded interleaving, as RFC0052.14 does: a test sink whose publish panics once the barrier is inside quiesce_publishes, asserting no install and no stamp and a failed outcome; and a record that panics the encode worker mid-batch, asserting the next barrier stamps nothing and a restart replays the remainder — run once with the barrier after the panic and once started concurrently with it, the seeded scheduler covering the store-before-decrement order, so a latch set after the count settles would fail the test rather than pass by timing. The retain path is already exercised by the existing skip-the-snapshot tests, so these extend them rather than duplicating.

• Truncation bounds (RFC0052.2) — an integration test building a WAL with several closed segments, a checkpoint above them and a snapshot floor deliberately below it, asserting exactly which files survive. The floor case is the important one: a test that only checks “segments disappear” passes on a bound that ignores the floor.

• Bounded growth (RFC0052.3) — the ourios-bench soak harness on its synthetic clock, asserting the WAL’s byte total and segment count stay bounded over a long run. This is the one criterion a unit test cannot express, because the defect is the absence of a periodic call; only elapsed cadence reveals it.

  The harness cannot do this today and has to be extended first. It builds a bare WAL and coordinator whose sampler only flushes the sink and runs compaction — there is no snapshot, checkpoint or housekeeping cadence in it at all, and its synthetic clock advances record timestamps rather than driving a timer. So the extension is part of this criterion’s cost, not an assumption behind it: the soak loop gains the §3.2 sequence on the synthetic clock, which is also the only way to reach a multi-hour backlog in a test.

• Rotation retry (RFC0052.4, RFC0052.5) — fault injection at each of the five rotation steps (the rename included), once-failing and always-failing, asserting recovery in the first case and a distinguishable terminal refusal in the second, plus Wal::open succeeding afterwards in both (the temporary-name property) and the temp files being swept. A proptest over which step fails and how many times keeps the five sites from being tested only one way.

• Pre-existing orphans (RFC0052.11) — a directory fixture hand-built in the shape today’s failure leaves, asserting that open still halts with OpenError::Corrupt and that the error names the file and the rotation-remnant shape. A fixture rather than fault injection, because the state predates the code under test.

• Capped passes (RFC0052.12) — a backlog well past the cap, asserting the per-pass unlink count and that repeated passes reach the uncapped end state; a leg seeds stale partials beside that backlog and asserts the first pass removes every partial and only cap-minus-partials segments. The concurrent-append half is asserted by timing out an append against an uncapped pass in a regression guard rather than by measuring wall clock, which would be flaky.

  These replace rfc0008_6_rotation_failure_quiesces_the_wal, whose “even after the underlying condition clears” assertion is the contract §3.3 changes. That replacement needs explicit approval (CLAUDE.md §6.2) and must keep both halves of what the original protected: no ack on an incomplete rotation, and a permanent refusal when the fault is persistent.

• Reclaim record (RFC0052.17) — directory fixtures for the four record states (absent, empty, valid with entries, corrupt) crossed with a restorable and an undecodable snapshot, asserting halt-or-pin per case and the Corrupt error’s file name; fault injection at the two crash points (before rename, before first unlink) through the same hook the rotation tests use; and a two-pass monotonicity leg reading the record back.

• Temp sweep (RFC0052.16) — a directory fixture holding all four file kinds (CHECKPOINT.tmp, RECLAIM, a *.snap.tmp and the partial; the record has no temp of its own), asserting exactly one is removed. A fixture rather than a live rotation, because the point is the selector, and the dangerous cases (CHECKPOINT.tmp, *.snap.tmp) are produced by other subsystems.

• Reclassification (RFC0052.15) — the classifier unit tests held on hold/794-wedged-classification, rewritten to the terminal-only rule and extended to assert the narrowness: an ordinary append or fsync I/O failure and a still-retrying rotation stay in the transient class while only the terminal rotation state leaves it; no arm carries a retry hint. Without that third assertion the test passes on a blanket removal of Retry-After, which would contradict RFC 0018 §3.2 rather than amend it.

• Floor pinning (RFC0052.13) — unit tests over the three RetainFloor cases, asserting a tenant without a snapshot pins the minimum at its oldest surviving frame, that segments below it holding only covered frames are reclaimed, that the pin lifts when its snapshot lands, and that the pinned case is not expressible as the no-consumer case. A churn leg writes one tenant once, snapshots it, and asserts it leaves the ledger when its last segment is unlinked. The startup leg uses a snapshots-root fixture whose directory fsync is made to fail (the file-in-place-of-directory technique from the rotation tests, since read-only permissions do not bind under root), asserting every tenant is pinned rather than any listed horizon used.

• Timer exclusion (RFC0052.14) — a seeded-interleaving test rather than a timing one: the window is narrow, and a wall-clock test that happens to pass proves nothing. Failing that, hold the timer artificially between the quiesce and the end of the cut while driving ingest, assert the submit blocks until the cut is captured and proceeds afterwards, and assert the records admitted after the cut are in neither the checkpoint nor the snapshot that barrier writes — the lock is never held across store I/O.

• Telemetry (RFC0052.7) — the in-memory metric exporter pattern already used for the ingest/sink instruments, asserting every name is in the exported stream; a runtime tracing assertion that drives each transition — terminal entry on the append path, floor pinned and lifted, latch set — and verifies event count, name and state, since the live-check validates definitions rather than emission; plus a weaver registry live-check pass over the new log events, since an event the tests never emit is an event the live-check never checks (that is how #795’s un-named event passed CI).

• No loss (RFC0052.10) — extends the existing SIGKILL crash-recovery test rather than adding a parallel one, with reclamation configured on a short cadence so the kill lands in the regime this RFC introduces. The #791 regression tests (refuse-then-resume with no append) move with the bound to RFC 0053.

Maturity, per docs/rfcs/README.md: green is every §5 criterion passing with the unit, property and corpus tests green — none of the live scenarios is optional. RFC0052.3’s recorded soak run and RFC0052.10 are additional incident-regression gates this RFC requires before validated, since those two demonstrate the incident cannot recur rather than that a unit behaves; they are not substitutes for the rest.

7. Open questions

• Whether §3.3’s count budget (three consecutive attempts) should become a time budget with backoff. A count is simpler to test; a time budget degrades better on a disk that is slow rather than broken.
• Whether the terminal rotation state also gets an operator verb to clear it without a restart, or whether a restart remains the documented recovery for a persistent durability fault.
• Whether disk_bytes should be computed per pass or cached. WalMetrics documents the directory walk as best-effort; exporting it on every collection may not be free on a WAL with many segments — though after §3.2 there should be far fewer.
• max_unlinks_per_pass’s default value only, which trades first-pass stall against how long a large backlog takes to clear. The knob is settled — §3.8 classifies it, with its class, proposed default and validated range — and the soak run (RFC0052.3) is what should confirm or move the number.
• Whether the actionable halt for a pre-existing rotation remnant (§3.3) deserves a WAL verb to remove the file, or stays a documented manual step. The decision must remain a human’s either way — no shape-based heuristic can tell rotation debris from real corruption — so this is about ergonomics, not safety.
• Whether a stale tenant floor blocking reclamation indefinitely should itself escalate (a second, louder state) or stay a visible metric an operator alerts on. §3.5 makes it visible; it does not decide.
• Whether to persist a published high-water mark with every flush, so that frames published between two barriers are not re-published on replay. Today that is RFC 0008/0014’s at-least-once contract above the checkpoint; this RFC narrows nothing there and bounds nothing there.

8. References

• Issue #791 — the incident: ingest wedged at 503, no self-recovery, no observability.

• Issue #793 — checkpoint/housekeeping called only from tests.

• Issue #796 — a publish unwind drops drained batches and the snapshot guard reads empty buffers as fully drained; specified by RFC 0053 §3.2.

• PR #794 — the OTLP Status body on ingest rejections (conformance only; the wedged classification was split out to hold/794-wedged-classification and lands with §3.3).

• PR #795 — counts a cadence panic (makes a dead sweep alertable; deliberately still stops, pending RFC0053.2).

• RFC 0053 — WAL backpressure and unwind safety; the stage that depends on this one, and which amends this RFC in these places:

  • RFC 0053 §3.2 amends §3.7’s Journal surface and this RFC’s sidecar layout: checkpoint takes a per-tenant published map, fn checkpoint(&mut self, durable_to: WalOffset, published: &HashMap<TenantId, PublishedMarks>) with PublishedMarks { records, audit } — a per-tenant pair, not a bare offset — persisted to a PUBLISHED sidecar written after CHECKPOINT in the same call, on the checkpoint’s own durable path, and joining §3.2’s sidecar matrix on the same fail-closed footing as RECLAIM.
  • RFC 0053 §3.2’s PUBLISHED has its own pinned layout under its own magic b"OWPB", sharing this record’s rules rather than being identical to it — the two-slot generation protocol, the per-slot dictionary, the fixed-stride arrays and the position-is-the-id rule — and keys on the same tenant key, so the dictionary and its u16 slot ids apply there too — the derivation has to be injective over the admissible tenant set, which a 16-byte digest of a 128-byte id is not — and its key field is sized to RFC 0048 §3.1’s 128-byte bound, which amends RFC 0046 §3.1 and is what MAX_TENANT_BYTES already encodes.
  • RFC 0053 §3.1 makes max_segments count closed retained segments, with the current segment and the segment an owed rotation creates outside the cap, so an owed rotation always has room.
  • RFC 0053 §3.2 adds Journal::publish_marks(&HashMap<TenantId, PublishedMarks>) — a PUBLISHED-only write with its own crash ordering, used by settlement and by the migration startup path — so checkpoint is not called merely to advance a publication watermark.
  • RFC 0053 §3.2 makes a permanently failed audit write a distinct outcome that refuses the dependent record publish, naming RFC 0005’s audit-sink contract; §3.1’s wait-or-write rule here covers only the transient case, which requeues.
  • RFC 0053 §3.2 defines the clear arm of §3.1’s packed failure word — which cut may CAS it back, and against which captured value — since stage 1 never clears and only a restart does. The packing itself is §3.1’s, adopted here so a same-epoch report cannot be lost between a reader’s loads.
  • RFC 0053 §3.2 adds an eligibility case §3.2’s predicate cannot express: a closed segment holding no valid frame — one its seal truncated back to a header — has no tenant offsets and no highest frame offset to compare, so it is unconditionally eligible and popped ahead of the horizon-driven candidates. That state cannot arise under this RFC alone, since rotation closes a segment only when it holds a frame, which is the assumption §3.2’s predicate rests on; it is RFC 0053’s sealing that creates it, so the case belongs there.
  • RFC 0053 §3.1 also amends §3.7’s append_batch: it takes a RotationDecision (Rotate / Reuse) produced once by the reservation under the journal mutex, so the age-sensitive rotation predicate is decided in one place rather than recomputed inside the append.
  • RFC 0054 §3 replaces the timer’s pre-cut guard — §3.2’s if failed_epoch <= barrier_epoch: skip — with proceed-and-decide, once its requeue makes the panicked batch’s records available to the next cut’s drain; the “only a restart clears it” wording in §3.1, RFC0052.1 and RFC0052.7 is amended with it, and RFC0054.4 asserts it. The unwind half of RFC 0053 moved there in the 2026-09-15 split.

• RFC 0008 §6.5 (rotation), §6.6 (recovery horizon), §6.7 (checkpoint and housekeeping), §6.8 (counters), §6.9 (tunables) — the mechanism this RFC supplies the policy for; §6.5’s durable-entry-first rule and Scenario RFC0008.6’s permanent refusal are superseded by §3.3 and RFC0052.4/.5. §6.7’s single-call Wal::housekeeping(retain_floor) is superseded by §3.2’s housekeeping_prepare / housekeeping_commit pair — a pass that records its plan before it unlinks cannot be one call. The single-call form survives behind a legacy-housekeeping feature that nothing but this crate’s own dev-dependency enables, so the RFC0008.7 tests keep their subject while the deployment invariant the legacy-root branch above rests on — that no production path calls it — becomes a property of the build rather than of a grep.

• RFC 0046 (the tenant-prefixed frame) — amended by §3.2:

  This RFC amends RFC 0046’s replay validation and criterion RFC0046.11 from a 256-byte tenant length to 128, the grammar RFC 0048 §3.1 pinned and which RFC 0046’s own resolved-questions note records as “the one tenant grammar every boundary applies at”. The amendment is a consistency fix rather than a new decision: the replay clause and RFC0046.11 were left at the superseded number, and the frame codec was left with them.

• RFC 0018 §3.2 (retryable error mapping) — amended by §3.3: its transient class lists “post-rotation quiesce”, which #791 disproved, and its two classes gain a third, server-terminal, client-retryable, for the terminal rotation state. RFC0018.3 stays satisfied, since the status is unchanged; only the class and the optional Retry-After move.

• RFC 0001 §6.9 — the miner snapshot high-water mark, and the hazard-#5 retain rule that makes it the truncation floor — amended by §3.2: horizons are per tenant (each tenant’s own last folded frame), the retain rule is tenant-aware and inclusive at a durably installed horizon (pins strict), and the stale-gap detector is restated on the RECLAIM record rather than on segment presence.

• RFC 0014 — the record sink and its flush triggers.

• CLAUDE.md §3.4 (WAL-before-ack), §3.6 (object storage is the source of truth), §6.3 (observability of ourselves).

• docs/hazards.md #3 (WAL durability versus latency), #4 (small files), #5 (template schema evolution across deploys — the retain rule).