Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

rfc: 0036 title: Write-side layout — compacted-partition clustering and row-group sizing status: red author: Jens Holdgaard Pedersen jens@holdgaard.org drafting-assistance: Claude created: 2026-07-21 supersedes: — superseded-by: —

RFC 0036 — Write-side layout

Status: red — reopened 2026-10-02. A regression after validated invalidated RFC0036.3’s memory criterion, so per docs/rfcs/README.md (Lifecycle: “a regression detected after Validated … reopens the RFC (if a criterion is invalidated)”) the RFC returns to red (issue #895). The §3.2 skip-spill gate measured encoded input bytes, so a ~15 MB / 1,555-file partition stayed under it yet decoded past 1.5 GiB and ran a node with a 1.5 GiB memory limit out of memory on every boot sweep. The reopened contract is the corrected §3.2 algorithm (one decoded-row budget governing the skip-spill decision and run formation) and the rewritten RFC0036.3 memory clause. The test that reopened it is many_small_inputs_within_the_encoded_budget_do_not_decode_at_once (crates/ourios-parquet/src/compaction/tests/bounded_sort.rs, PR #900). RFC0036.3 now has two memory clauses: the decoded-row budget (#900) and the byte-bounded phase-2 merge (#916). The RFC returns to green once both have landed and every §5 criterion passes on main; to validated once the compaction benches are re-recorded and the docs/benchmarks.md §7 thesis-gates pass on representative corpora; and to accepted only on the maintainer’s sign-off (docs/rfcs/README.md lifecycle). The status note immediately below is the pre-reopen history (the earlier validated/accepted record); the rest of this document, including the revised §3.2 algorithm and RFC0036.3, is the current contract.

Status note. accepted (2026-07-22, maintainer sign-off — the terminal state), amended 2026-07-22 with the §3.3 adaptive compacted-threshold (target-K): the fixed 32 MiB threshold was proven inert on real v8 (§9.28/§9.29 — a real hour compacts to one group and prunes nothing) yet fragments large hours, so it is replaced by clamp(input_total / 8, 1 MiB, 32 MiB) (accepted RFCs may be amended — docs/rfcs/README.md; maintainer-approved 2026-07-22). §9.30 measures the win at the adaptive default (7.61× real-corpus materialization-bytes win, 20 of 21 groups pruned, where the fixed 32 MiB default had skipped).

Reached validated on all five §5 scenarios green plus the comparative evidence below. RFC0036.1 (footer inspection — compacted threshold, sorting_columns, per-group service min/max — plus the §6 merge property), RFC0036.3 (D3 file-band + forced-spill memory bound), RFC0036.4 (shuffled-listing byte-identity rebuild), and RFC0036.5 (pre-/post-RFC read-path parity, no schema migration) pass in crates/ourios-parquet/tests/it/rfc0036_write_side_layout.rs (and compaction.rs’s forced-spill unit test); RFC0036.2 (window-query scanned-row-group bound) passes in crates/ourios-querier/tests/it/rfc0036_window_materialization.rs. The external merge sort (run formation → capped-fan-in k-way merge), the §3.3 adaptive threshold (adaptive_flush_bytes), and the §3.4 sorting_columns declaration are in compaction.rs/writer.rs. Design review is done — maintainer go, 2026-07-21. The §7 decisions needed to ship this slice are recorded below (the threshold question is now resolved by the §3.3 adaptive model; the authoritative ceiling/target-K sweep against the L1/L3 curve stays deferred to the ourios-bench harness); the remaining §7 questions stay open as noted there.

validated evidence (2026-07-22). The deferred comparative arm resolved in two parts, both recorded in docs/benchmarks.md:

  • RFC0036.5 no-regression (§9.26, authoritative baseline-8vcpu-32gib, HEAD 5e5aa66). The frozen RFC 0031 dispatch reran on post-RFC-0036 main: every measured frozen gate passes in §9.24’s band (L1 ~99×, L2 ~43× + floor 1.40, L4 ~85× + floor 3.62, L6 k=2000 latency 4.0). Finding: the comparative harness writes one ingest file per partition, so compact_partition no-ops and RFC 0036’s sort never runs there — every ourios_bytes_read is byte-identical to §9.24. So the baseline run proves no regression but structurally cannot show RFC 0036’s win; a multi-file-per-partition harness that re-bases the frozen-gate store is future work, not a validated blocker (RFC0036.2 bytes is a §2.2 diagnostic, not a gate).
  • RFC0036.2 materialization before/after (§9.27, in-repo, deterministic). Measured directly on a genuinely-compacted store: the sort takes a one-service window from materialising the whole file (unsorted: 1/1 groups, 100.5 MB — no row group prunes) to a contiguous minority (sorted: 2/6 groups, 70.0 MB) for the identical answer — a 1.43× materialization-bytes win, modest and honest (the compacted file is ~2× larger on disk; §2.2’s registry floor is why the gate is the scanned-row-group bound, which rfc0036_2_window_materialization_bound enforces).

The scanned-row-group gate was always the load-bearing acceptance criterion (RFC0036.2 §5, green in-repo); the v8-corpus comparative materialization number remains desirable future work behind the harness change above.

How to read this document. This is the write-side layout lever that docs/benchmarks.md §9.13 named and §9.24 left “parked on its own line” — hazard #4’s layout fork, carried from the #498 scoreboard. The comparative program measured why time-window browses lose the storage-bytes channel to Loki (§9.13 runs #8–#17): a compacted hour is one file whose row groups rotate only at 128 MiB uncompressed and declare no sort, so a one-service window query materializes essentially the whole hour. This RFC clusters rows at compaction time by (promoted service.name, time_unix_nano), rotates compacted row groups at a smaller threshold, and declares Parquet sorting_columns — ingest is untouched, no Parquet schema change, store-build determinism preserved. The framing is deliberately honest: the L6 storage-bytes channel stays a published diagnostic (RFC 0031 §5 declined to gate it, and the ~188 KB layout-independent template-registry acquisition is its floor — §2.2). The goal here is to collapse the materialization term — a window query over one service in one hour should fetch a few small row groups, not the whole hour — not to beat a floor that layout cannot touch.

1. Summary

Post-compaction, a (tenant, hour) partition is one file (compaction.rs outputs a single consolidated file per partition) whose row groups rotate only when the writer’s uncompressed in-progress buffer crosses 128 MiB (writer.rs:60), whose rows sit in ingest/append order, and which declares no sorting_columns (none exist anywhere in the codebase). On the v8 comparative corpus that yields roughly one row group per hour holding all services (§9.13 run #17), so neither the time min/max statistics nor the RFC 0022 promoted service.name bloom can skip anything — the measured mechanism of the L6 window-browse loss. This RFC changes the compacted layout only: (1) compaction sorts the partition’s rows by (promoted service.name, time_unix_nano) via a bounded-memory sort-run merge whose peak residency is a decoded-row budget plus one input file (§3.2, reopened 2026-10-02); (2) compacted row groups rotate at a smaller, adaptive threshold that scales with partition size (§3.3 — clamp(input_total / 8, 1 MiB, 32 MiB), amended 2026-07-22), giving time/service pruning its granularity back on hours of any size; (3) the writer declares Parquet sorting_columns so order-aware readers can exploit the layout. No schema change, no migration, no ingest-path change, and byte-identical store rebuilds are preserved. The explicit non-goal: winning the L6 storage-bytes channel, whose floor is the layout-independent registry acquisition (§2.2).

2. Motivation

2.1 The measured mechanism

docs/benchmarks.md §9.13 published the L6 window-browse loss honestly and diagnosed it precisely. On a browse-k-rows query Loki reads only the tiny chunk slice its label stream + time index point at (16,250 B storage-side at k=100), while Ourios pays fixed per-query costs that dwarf a k-row answer: 1,931,911 B at k=100 on the authoritative run (§9.24). Run #17’s diagnostic sharpened the why: scoping the same window to the lowest-volume service (“ad”) improved Ourios only ~22% — no bloom collapse — because v8’s hour partitions each hold roughly one row group containing all services, so the promoted service.name bloom has nothing to skip and the time min/max spans the whole hour. §9.13’s closing assessment named the fix: “the tier-changing lever is write-side layout (service clustering / row-group sizing — hazard #4 territory, an RFC-level change), not query-side tuning.” §9.24 repeated it: the write-side lever “stays parked on its own line.” This RFC is that line.

The layout facts, verified in code:

  • Ingest seals a partition buffer at a 256 MiB estimate target, a 1 GiB total ceiling, or 300 s of age (ourios-server/src/receiver.rs:55–57; these are RFC 0014 §3 defaults, not yet RFC 0004 knobs — RFC 0014 §7).
  • The writer flushes a row group when ArrowWriter::in_progress_size — an uncompressed estimate — crosses 128 MiB (writer.rs:60, checked at writer.rs:627 and writer.rs:644). There is no max_row_group_size property and no sorting_columns declaration anywhere; rows are in append order.
  • PartitionKey is (tenant, year, month, day, hour) only (partition.rs:29–43) — no service dimension in the path.
  • Compaction (compaction.rs:184–308) streams inputs one file at a time (peak decoded memory = one input file — a deliberate, commented property, compaction.rs:228–234), appends them in input order into one output file per partition, and rotates row groups at the same 128 MiB threshold. §9.7’s band-scale D3 output was a single 456.7 MiB file.

2.2 What layout can fix, and the floor it cannot

Two distinct terms make up an Ourios window query’s bytes:

  1. The registry acquisition. Every body-rendering query pays the RFC 0033 template-map acquisition — the warm v2 artifact is 187,904 B (RFC 0033 §9; 187,906 B on the §9.24 authoritative run). This term is layout-independent: it is paid before any row group is touched, and for k=100 it alone exceeds Loki’s entire 16,250 B read by ~11×. No write-side layout changes it.
  2. The materialization term. Everything else — the column chunks of the row groups that survive pruning. At k=100 on §9.24 this is roughly 1,931,911 − 187,906 ≈ 1.74 MB: the whole hour, because one all-services row group survives pruning by construction.

This RFC targets term 2 only, and says so plainly: layout cannot beat the ~188 KB registry floor and does not try. Accordingly the L6 storage-bytes channel stays exactly what RFC 0031 made it — a published diagnostic, not a gate (RFC 0031 §5 declined to gate it; the §7 frozen L6 gate is the latency floor, which already passes: 0.370 / 4.341 on §9.24). Illustrative arithmetic, not a promise: with the hour clustered by (service, time) and rotated at the §3.3 adaptive threshold, a one-service k=100 window’s answer lives in one or two small row groups; the materialization term drops from ~1.74 MB toward the size of those chunks’ matched columns, leaving total bytes dominated by the registry constant. The §5 criteria gate the mechanism (row groups fetched), and the before/after bytes are measured and published in the §9 series as the diagnostic they are.

2.3 Why compaction is the layer

The ingest path just got its concurrency model rebuilt (RFC 0035: streamed append, order-insensitive Parquet encode on a bounded pool, an encode-drain-and-flush barrier keyed to WAL rotation). Sorting at ingest would sit directly on that fresh machinery and on the ack path (§4). Compaction, by contrast, already rewrites every row of the partition — it is the RFC 0022 §3.4 re-projection point, where history converges toward the current promoted attribute set — so clustering rides a pass that exists, off the hot path, with its cost bounded by the compaction cadence. Hazard #4’s mitigation already owns this layer (“background compaction job per tenant; cadence is a tunable”); this RFC extends what that job does, not where work happens.

3. Proposed design

3.1 The clustering key

Compacted rows are ordered by, in precedence:

  1. Promoted service.name value, lexicographic byte order of the UTF-8 string, absent/null first. Lexicographic, not first-seen or dictionary-ordinal order: first-seen order depends on ingest interleaving and would break rebuild determinism (§3.5). Tenants whose promoted set does not include service.name fall back to key 2 alone — a time-only sort still buys row-group time-pruning granularity (§7).
  2. time_unix_nano, ascending.
  3. Deterministic tie-break — (input-file ordinal in sorted-basename order, row ordinal within that input). Never exposed as a declared sort; it exists so equal-key rows have one canonical order and the output is byte-identical across runs and listing orders (§3.5). Compaction already sorts input basenames for its audit event (compaction.rs:267–268); the tie-break reuses that order.

3.2 Bounded-memory sort: run formation + k-way merge

Reopened 2026-10-02 (issue #895). This section originally formed one sorted run per input file and let the whole partition skip spilling while its encoded input total stayed ≤ 256 MiB, arguing that one seal-target’s worth of input decodes no larger than one worst-case input file. Production invalidated that: small, well-compressed inputs decode to many times their encoded size, so a ~15 MB / 1,555-file partition stayed under the gate yet decoded past 1.5 GiB, and the boot compaction sweep ran a node with a 1.5 GiB memory limit out of memory on its first candidate, on every start. The algorithm below is the corrected contract: one decoded-row budget governs both the skip-spill decision and run formation.

A plain k-way merge of the input files cannot produce this order: the key leads with service.name, which append order does not cluster at all, and RFC 0035 explicitly disclaims intra-partition row order on ingest-side files (RFC0035.5 — concurrent encode may reorder rows; “near-time-ordered” is an observation, not an invariant). The design is therefore a textbook external merge sort with a decoded-row budget B (in_memory_max_bytes, default 64 MiB). B is measured as an estimate of the heap each decoded row holds — the record, its strings, params, separators and attribute trees — and never as encoded input bytes.

  1. Run formation. Read the inputs in sorted-basename order (the §3.1 input ordinal), each streamed batch by batch, appending their decoded rows to one buffer in (input ordinal, row ordinal) order. Each row is validated against the partition as it decodes (RFC0009.5). At the end of each input, if the buffer’s decoded footprint exceeds B, stably sort the buffer by the §3.1 key and spill it to local scratch as one sorted run, then empty the buffer (local disk is cache, not truth — CLAUDE.md §3.6; the run format is §7). Runs are cut only between inputs, so a run holds one or more whole inputs and never part of one, and runs are numbered in spill order.
  2. Skip-spill. If no run was spilled by the last input, the whole partition fits B: sort the buffer in place and write it straight to the output. Otherwise, if the buffer still holds rows (inputs read after the last spill that kept it at or below B), stably sort it by the §3.1 key and spill it as the final run; an empty buffer spills nothing, so no run is empty. Then go to step 3 with the runs spilled so far. Every run is therefore sorted before it is merged.
  3. Merge. Stream a k-way merge over the sorted runs, holding one decoded batch per run (the Reader wraps the streaming ParquetRecordBatchReader), breaking §3.1 key ties by run ordinal. The fan-in cap F (64) bounds the number of runs merged at once: while more than F runs remain, merge consecutive groups of F runs into intermediate runs, preserving run order, and repeat. Each run’s reader batch is sized in bytes, not rows: every run records the largest per-row decoded footprint it holds (its widest row, measured as it is formed, or the maximum over the runs a hierarchical merge combines). A merge of k runs gives each reader ⌊B / (k + 1)⌋ bytes, divides that share by the run’s widest row, and clamps the result to [1, 1024] rows. The 1,024-row cap is the batch narrow rows always used. The one-row floor exists because a merge cannot advance a run without holding its head row (#853). Merged rows collect in one output chunk of 1,024 rows (the writer’s sub-batch), which is emitted into the existing Writer (rotating row groups at the §3.3 threshold) each time it fills.

Order and determinism. The buffer holds rows in (input ordinal, row ordinal) order, every sort is stable, runs are contiguous in that order, and the merge breaks ties by run ordinal, so the §3.1 total order holds however the inputs are grouped into runs. Both the skip-spill and the merge path drive the Writer with the same sub-batch sequence, so their output is byte-identical (§3.5); spilling changes only local scratch I/O.

Memory bound (the load-bearing claim). Phase 1 holds at most B plus one decoded input: the buffer is under B when an input starts, and that input is the most it can overshoot by. Phase 2 holds, whatever the run count, the open runs’ reader batches plus the output chunk. The reader batches together hold at most max(B, k × widest row) decoded bytes for k open runs. That is B, plus at most one row per run whose widest row is wider than its ⌊B / (k + 1)⌋ share: such a run holds its one head row, the documented overshoot. The output chunk holds at most 1,024 rows, a fixed term the writer’s sub-batch contract sets. In rows, phase 2 still holds at most (F + 1) × 1,024. The phases do not overlap, because the run-formation buffer is released before the merge starts. Neither bound depends on the partition’s file count or compression ratio. The writer’s in-memory output accumulation (ArrowWriter<Vec<u8>>) is unchanged. Everything around the sort — manifest bootstrap, CAS commit, GC, the RFC0009.5 per-row partition validation — is untouched.

3.3 Compacted row-group threshold — adaptive (target-K)

Amended 2026-07-22. This section originally shipped a fixed 32 MiB compacted threshold. §9.28/§9.29 proved that value is inert on the real v8 corpus: a real per-hour partition is only a few MiB compressed — far below 32 MiB — so it compacts to a single row group and prunes nothing (§9.29 skipped at the 32 MiB default), while the same fixed value fragments large hours. The threshold is therefore replaced by an adaptive one that scales with partition size, so a partition of any size lands on roughly a target group count. The fixed value is retained as the ceiling. Maintainer-approved design decision, 2026-07-22.

Compacted output rotates row groups at an adaptive, per-partition threshold (ingest-side files keep the fixed 128 MiB ROW_GROUP_FLUSH_BYTES):

adaptive_flush_bytes = clamp(
    estimated_output_bytes / TARGET_COMPACTED_ROW_GROUPS,
    MIN_COMPACTED_RG_BYTES,
    MAX_COMPACTED_RG_BYTES,
)
  • TARGET_COMPACTED_ROW_GROUPS = 8 — the group-count target: enough to cluster services and give pruning granularity, not so many the per-group footer/page-index overhead dominates.
  • MIN_COMPACTED_RG_BYTES = 1 MiB — the floor. It clamps the computed rotation threshold (estimate / K) up to at least 1 MiB, so compaction never rotates pathologically often on a small partition. It bounds the threshold, not the resulting group size: a partition that never crosses the threshold is a single row group, the final remainder group can itself be < 1 MiB, and whether the partition rotates at all depends on the writer’s in_progress_size crossing the threshold — not a fixed size rule. Its role is that a small-but-not-tiny hour — a few MiB compressed — rotates into several groups instead of one, the lever that makes small real-v8 hours prunable (§9.29/§9.30).
  • MAX_COMPACTED_RG_BYTES = 32 MiB — the ceiling (the old fixed COMPACTED_ROW_GROUP_FLUSH_BYTES value). A huge partition gets more than K groups, each capped at 32 MiB, so a compacted row group never grows back toward the 128 MiB ingest band.
  • estimated_output_bytes is the sum of the partition’s live input file sizes, read from the store during compaction (the inputs are already listed in compact_sorted). The sorted output is the same rows re-compressed — ≈ or a little smaller — so the input total is a safe upper estimate. The function is deterministic in its input, so the same partition compacts to byte-identical output (§3.5 / RFC0036.4).

Worked examples: a 14 MB hour → 14/8 = 1.75 MB flush → ~8 groups (pruning works); a 400 MB hour → 400/8 = 50 MB, capped to 32 MB → ~13 groups, each ≤ 32 MB; a 4 MB hour → 4/8 = 0.5 MB, floored to 1 MiB → ~4 groups (instead of the single group the fixed 32 MiB gave).

Precedence. An explicit compact_partition_with_flush_threshold argument (the §7 sweep seam / tests) wins over the OURIOS_COMPACTED_RG_BYTES env override (the operator escape hatch), which wins over the adaptive value. Production sets neither and gets the adaptive threshold.

Rotation fires on ArrowWriter::in_progress_size, the writer’s estimate of the buffered row-group bytes (dominated by already-encoded page data, not raw uncompressed input), so on-disk row-group size tracks the threshold closely. Combined with the §3.1 sort, each row group’s service.name min/max spans one service — or a boundary pair, or (for a service smaller than one row group, wedged between two others) that service plus its two neighbours — while its time min/max stays tight within a service. A query for any single service scans only the group(s) whose min/max contains it and prunes the rest on plain footer statistics, without even needing the bloom.

This amends hazard H4’s row-group band. docs/hazards.md H4 targets “row-group size 128 MB – 1 GB”; this RFC deliberately drops compacted row groups below that band. The band’s purpose is file economics — LIST calls, footer reads, cold-cache hits — and those are governed by the file band (256 MiB – 2 GiB), which is untouched: compaction still emits one file per partition, D3 still measures files. Within one file, a smaller row group costs a few more footer metadata entries (~14 vs ~4 at D3 scale) and buys pruning granularity — the trade this RFC exists to make. On acceptance, H4’s mitigation bullet is reworded to scope the 128 MB – 1 GB row-group target to ingest-side files and state the compacted threshold as the pruning-granularity knob (a one-line docs/hazards.md edit shipped with the implementation; H4.4’s detection signals are file-based and unaffected).

3.4 Declared sorting_columns

The compaction writer declares Parquet sorting_columns — the §3.1 keys 1 and 2 (or key 2 alone for time-only tenants) — via WriterProperties. Two honest clarifications: (a) the pruning win of §3.3 comes from the physical clustering making per-row-group statistics tight, not from this metadata — statistics prune whether or not a sort is declared; (b) the declaration is what lets order-aware execution (DataFusion sort-elision, merge scans, future DSL ORDER BY/limit pushdown) trust the layout without a defensive re-sort. It is pure footer metadata: no Parquet schema change, old files without it read exactly as before, readers that ignore it are unaffected — CLAUDE.md §3.5 is satisfied with no migration (RFC0036.5 pins this). Ingest-side files declare nothing in this RFC (their rows are genuinely unsorted post-RFC 0035; declaring a sort they don’t have would be a lie — a seal-time sort that would make a time-only declaration true is §7).

3.5 Determinism

The comparative harness depends on byte-identical store rebuilds (§9.13’s determinism note: “for repeated measurements of the same build and configuration, Ourios’s bytes are byte-identical” — that property is what lets the run series read as an optimisation ledger). The §3.1 key is a total order over the partition’s rows: lexicographic service value + timestamp + the (sorted-basename input ordinal, row ordinal) tie-break leave no two rows unordered, so the merged row sequence is a pure function of the input files’ contents and names. Row order alone does not imply byte identity, though — page and row-group boundaries, dictionary state, and footer contents must also be deterministic. They are, by the same writer-level invariants today’s §9.13 property already rests on: fixed sub-batching (SUB_BATCH_ROWS), a fixed row-group threshold evaluated on the same deterministic in_progress_size accounting, fixed writer properties (codec level, dictionary/statistics/bloom settings), and no time-or-randomness-dependent metadata. Deterministic rows fed through a deterministic serializer yield deterministic bytes — the identical argument that makes today’s unsorted builds byte-identical, with the sort adding only a deterministic permutation and (for spilled runs) deterministic run boundaries from fixed spill thresholds. RFC0036.4 pins the end-to-end claim with a byte-identity rebuild test, which subsumes the row-order property.

3.6 What stays exactly as-is

The ingest path in full — streamed append, the RFC 0035 encode pool and its drain-and-flush barrier, WAL-before-ack, the flush policy constants. The Parquet schema and every column’s encoding. The partition path scheme (no service dimension — §4 rejects it). The compaction manifest protocol (bootstrap, CAS, GC, orphan sweep) and the RFC 0022 re-projection semantics — clustering rides the same rewrite pass and re-projects under the same current promoted set. The query DSL and querier surface. The L6 gate disposition: the latency floor stays the frozen gate, the storage channel stays a published diagnostic.

4. Alternatives considered

  • Service sub-partitioning (tenant × time × service paths). Adding service.name to PartitionKey would make pruning trivial — and reintroduce exactly the hazard this RFC lives under: per-service files multiply file counts by service cardinality, and low-volume services (v8’s “ad” at ~34 s of activity per window) produce precisely the small-file/LIST blowup H4 exists to prevent. It is also a physical path layout change — every existing partition would need rewriting or dual-path read logic, a migration burden §3.5 reserves for schema-level necessity. Row-group-level clustering buys most of the pruning at zero path/migration cost. Rejected.
  • Ingest-time sorting. Sort rows before the ingest-side writer instead of at compaction. This breaks the streamed-append model: a sort needs the partition’s rows resident and re-orderable, so buffers pin their contents until seal, inflating residency against the 1 GiB SINK_CEILING_BYTES and adding latency-shaped work to the path RFC 0035 just relieved — and it tangles directly with the fresh encode pool, whose correctness argument (order-insensitive emit) was accepted weeks ago. Ingest-side files are short-lived (compaction consumes them); sorting them buys granularity only until the compactor runs. Rejected in favour of sorting where rows are already rewritten.
  • Gating the L6 storage channel. Set a storage-bytes floor and drive layout work against it. RFC 0031 §5 already considered and declined this — and §2.2 shows why it is unwinnable as a gate: the registry acquisition alone exceeds Loki’s entire k=100 read, independent of layout. Gating it would either force dishonest accounting (exclude the registry) or freeze a guaranteed FAIL. The channel stays a published diagnostic; the mechanism gets the gate (RFC0036.2). Rejected.
  • Do nothing. The frozen gates all pass (§9.24) — no gate forces this work. But §9.13 and §9.24 both name write-side layout as the remaining storage-side lever, window materialization is the honest weak spot the comparative program documented, and hazard #4 explicitly escalates layout tuning to an RFC. Leaving the named lever unpulled leaves every window query paying whole-hour materialization for a k-row answer. Rejected.

5. Acceptance criteria

Scenario RFC0036.1 — compacted layout (clustering + sizing + declaration).

  • Given a partition holding ≥ 2 input files whose rows span multiple promoted service.name values and interleaved times
  • When the partition is compacted
  • Then footer inspection of the consolidated file shows: row groups rotated at the configured compacted threshold (each uncompressed size ≤ threshold + one sub-batch’s bounded overshoot), sorting_columns declared as §3.1 keys 1–2 on every row group, and per-row-group service.name min/max spanning at most a boundary pair of services
  • And decoding the file yields rows in §3.1 key order, with the row multiset equal to the inputs’ union.

Scenario RFC0036.2 — window-query materialization (the point).

  • Given a compacted store built from a §9-style corpus (the v8 shape: one hour, many services, promoted service.name)
  • When the L6-shape query (one service, k-row time window) runs
  • Then the row groups scanned (the RFC 0016 scanned/pruned counts) are ≤ ceil(B_sw / T) + 2, where B_sw is the queried service’s bytes within the window (measurable from the compacted file’s footer: the sorted layout places one service’s window in contiguous row groups) and T is the configured row-group threshold — i.e. the groups that hold the answer plus at most two boundary groups, not the whole hour
  • And the before/after materialization bytes (total minus the registry acquisition) are measured on the same corpus and published in the §9 series as the storage-channel diagnostic — expected to fall by roughly the row-group-count ratio; the gate here is the scanned-row-group bound, not a bytes ratio (§2.2 — the registry floor makes bytes a diagnostic).

Scenario RFC0036.3 — compaction properties preserved (D2 / D3 / memory).

  • Given the §9.7-scale compaction workload (band-scale partition, tens of input files)
  • When the sorted compaction runs
  • Then D3 holds unchanged (one output file per partition, inside the 256 MiB – 2 GiB band, < 5% of live files below 128 MiB) and D2 compaction throughput stays within an agreed band of the §9.7 measure (sorting is not free; the band is set at red from a first measurement, and “keeps up” — throughput ≫ per-partition seal rate — must still hold)
  • And (reopened 2026-10-02) for any N-file partition — including many small inputs whose encoded total is far below the budget B while their decoded rows exceed it — peak decoded-row residency stays within the phase-specific bounds: at most B plus one decoded input in phase 1, and (F + 1) × one decoded batch (F open runs plus the merge’s output chunk) in phase 2. Phase 2 is also bounded in decoded bytes, whatever the row width: the open runs’ reader batches hold at most max(B, k × widest row) for k open runs (B, overshot by at most one row per run whose widest row exceeds its ⌊B / (k + 1)⌋ share), plus the 1,024-row output chunk (#853). Holding the whole partition is allowed only within those bounds (the skip-spill case, where it fits B); the gate is measured on decoded rows, not encoded bytes. B is a byte budget over the per-row decoded-footprint estimate, so a row-counting test states the phase-1 bound in rows as B divided by the smallest per-row footprint in the fixture, plus one input’s rows.

Scenario RFC0036.4 — determinism (the harness’s contract).

  • Given the same set of input files (same bytes, same names)
  • When the partition is compacted twice — including with the store returning listings in different orders
  • Then the two consolidated outputs are byte-identical, preserving the §9.13 determinism property the comparative ledger depends on.

Scenario RFC0036.5 — no read-path or schema regression.

  • Given stores built before and after this change, and old (pre-RFC) compacted files read by the new reader
  • When B1/B2 and the frozen RFC 0031 comparative gates run against the post-change store
  • Then every frozen gate still passes with the L1/L3/L4 pairs not degraded beyond the documented Loki-wobble band (sorted, smaller row groups should help or be neutral — measured, not assumed), query results are identical row-sets, and old files (no sorting_columns, 128 MiB row groups) read without error or special-casing — no migration exists because none is needed (CLAUDE.md §3.5).

6. Testing strategy

Mapped to CLAUDE.md §6.2; techniques per §5 scenario id:

  • RFC0036.1 — footer-inspection unit tests in ourios-parquet: compact a synthetic multi-service partition, then assert via ParquetMetaData the row-group sizes, sorting_columns, and per-group service.name/time statistics; decode and assert §3.1 order. Plus a property test (proptest) for the merge itself: arbitrary input files with arbitrary service/time/duplicate-key mixes ⇒ output multiset equals input union, output is §3.1-sorted, and equal-key rows land in tie-break order (the stability clause RFC0036.4 leans on).
  • RFC0036.2 — the comparative dispatch + querier counters. The L6-shape pair on the v8 corpus through the RFC 0031 harness; assert the scanned/pruned row-group counts (RFC 0016 emits them raw) against the ceil-bound; record before/after bytes in the §9 series. A smaller in-repo integration test pins the scanned-count bound on a synthetic hour so CI catches granularity regressions without the full harness.
  • RFC0036.3 — criterion compaction bench + a memory test. The existing compaction bench group re-run with sorting to set and then hold the D2 band; D3 assertions unchanged (rfc0009_1_*-style structural tests extended). Memory: compact an N-file partition under a decoded-row residency gauge and assert residency stays within the phase-specific bounds, in rows: phase 1 holds at most ⌊B / f⌋ rows plus one input’s rows, where f is the smallest per-row decoded-footprint estimate in the fixture (the same decoded_footprint the sort charges against B), and phase 2 at most (F + 1) × one batch’s rows. The phases do not overlap, so a single peak is checked against the larger of the two bounds. The skip-spill fixture may hold the whole partition, since it fits B. Reopened 2026-10-02: the gauge test includes a many-small-inputs partition whose encoded total is well below its budget (B set to twice the encoded total) while its decoded rows exceed that budget (many_small_inputs_within_the_encoded_budget_do_not_decode_at_once), which fails on the encoded-bytes gate. Bytes (#853): the same gauge also counts decoded-footprint bytes. wide_rows_merge_within_the_byte_budget compacts a hierarchically merged partition of wide rows (40 attributes × 250 B) under a small B and asserts that peak decoded bytes stay within the larger of the phase-1 bound (B plus one input) and the phase-2 bound (max(B, F × widest row) plus 1,024 widest rows for the output chunk). It fails when reader batches are sized in rows. cursor_batches_are_sized_from_the_byte_budget pins the sizing, including the one-row floor and the 1,024-row cap, and byte_sized_merge_batches_write_the_in_memory_bytes pins §3.5 byte identity under one-row merge batches. All three are in crates/ourios-parquet/src/compaction/tests/bounded_sort.rs (#916).
  • RFC0036.4 — a rebuild differential. Compact the same inputs twice — second run with a shuffled listing order (store fake) — and assert byte equality of the outputs (a file hash is correct here, unlike RFC0035.5’s decoded-row equality: byte identity is exactly the property claimed).
  • RFC0036.5 — existing suites + the comparative gates. B1/B2 and the frozen-gate dispatch on a post-change store; the RFC 0005 reader forward/backward tests extended with a pre-RFC-0036 fixture file (no sorting_columns) to pin no-migration reads.

7. Open questions

  • The compacted row-group threshold — RESOLVED 2026-07-22 by the §3.3 adaptive amendment (target-K). Originally settled at a fixed 32 MiB; the amendment replaces it with adaptive_flush_bytes = clamp(estimated_output_bytes / 8, 1 MiB, 32 MiB), keeping the 32 MiB value as the ceiling. Why the fixed value had to go: §9.28/§9.29 proved it inert on the real v8 corpus — a real per-hour partition is a few MiB compressed, so at 32 MiB it compacts to one row group and prunes nothing (§9.29 skipped at the 32 MiB default), while the same fixed value fragments large hours. The adaptive floor (1 MiB) is what rotates a small real hour into several service-clustered groups; the ceiling (32 MiB) caps huge hours. §9.30 measures the win at the adaptive default (no longer inert). The scanned-count gate (RFC0036.2) does not hard-code a threshold: it recomputes T = adaptive_flush_bytes(input_total) — exactly what the writer derived — and the bound ceil(B_sw / T) + 2 from it, so the gate moves with whatever the layout produces. It is not auto-satisfied — the gate still fails if the target service stops clustering contiguously and extra groups are scanned. The indicative in-repo sweep is done (§9.28, local M-series, synthetic-compressible, service-clustered corpus — the shape §9.27’s random payload was not); the authoritative 16/32/64 MiB sweep against the L6-shape scanned-bytes curve and L1/L3 neutrality (more row groups = more footer entries and per-group index overhead) stays a paid baseline-8vcpu-32gib measurement deferred to the baseline harness and not a green blocker. §9.28 findings (2,160,000-row six-service hour, 7.20× compression): (1) the compacted file does not grow as the threshold shrinks — 32 and 64 MiB give a byte-identical file, 16 MiB is larger by only +0.80% — so §9.27’s “compacted file ~2× larger” was a random-bytes artifact, not a real-log property; (2) a fixed one-service window materialises half the bytes at 16 MiB (14.66 MiB) vs 32/64 MiB (28.94 MiB), same answer, for that +0.80% disk cost — the pruning-granularity trade is good on compressible data and improves as T shrinks; (3) 32 and 64 MiB are near-identical because arrow’s default 1,048,576-row group cap (no max_row_group_size is set) fills a group at ~30 MiB on this ~30 B/row corpus and so trips before either byte threshold — a granularity floor finer than 32/64 MiB. It is not a bug to “fix” by raising the cap: doing so would let 32/64 MiB coarsen (~2 groups), the wrong way for pruning. The pruning lever is a smaller byte threshold (16 MiB, byte-governed → finer groups), the §7 sweep question — not the row cap. Disposition (superseded 2026-07-22): these indicative numbers showed the trade leans toward smaller thresholds and, together with §9.29’s finding that a fixed 32 MiB is inert on real per-hour v8 volume, motivated the §3.3 adaptive amendment: the threshold now scales as input_total / 8, floored at 1 MiB and capped at 32 MiB, so small hours get the finer groups §9.28 favoured (down to the 1 MiB floor) and large hours stay capped. §9.30 confirms the win at the adaptive default. The authoritative full-v8 sweep of the ceiling and target-K against L1/L3 neutrality stays deferred to baseline-8vcpu-32gib. Still tunable via the OURIOS_COMPACTED_RG_BYTES env knob (overrides the adaptive value) and the explicit compact_partition_with_flush_threshold seam (the sweep’s test arm; setting a process env var is unsound under cargo test’s parallelism).
  • Sort-key stability definition — settled as lexicographic. The §3.1 key is lexicographic service.name, not first-seen/dictionary ordinal (first-seen is interleaving-dependent and would break RFC0036.4). Confirmed: dictionary encoding of the compacted service.name column is unaffected by row order (it keys on the set of distinct values, not their arrival order), and RFC 0022 bloom sizing is per-value, not order-dependent — nothing downstream prefers first-seen order.
  • Unpromoted-attribute tenants. No promoted service.name ⇒ time-only sort (§3.1). Still wins time-pruning granularity; confirm the sorting_columns declaration degrades to the single time key cleanly and RFC0036.1’s assertions have a time-only variant.
  • Ingest-side time-only sorting_columns. Ingest files are near-time-ordered but not sorted (RFC0035.5), so declaring order today would be false. A seal-time sort of the in-memory buffer by time_unix_nano is cheap (the buffer is already resident, ≤ the 256 MiB target) and would make a time-only declaration true — a separate small win for queries that hit not-yet-compacted files. Possibly in-scope at red if it falls out of the writer work; otherwise its own follow-up.
  • Interaction with RFC 0022 re-projection. Clustering rides the same compaction pass that re-projects promoted columns (§3.6). Confirm a promoted-set change between builds is correctly out of scope for RFC0036.4 (determinism is claimed for same-configuration rebuilds — §9.13’s phrasing — not across config changes), and that sorting keys read the current promoted set, matching re-projection.
  • Run format and fan-in cap F. Decided: sorted runs spill as Parquet in the data schema with spill-oriented properties (no dictionaries, no statistics, ZSTD-1), reusing the existing Reader/writer rather than a parallel Arrow-IPC codec path — a run’s bytes influence the output only through its decoded rows, so IPC’s cheaper encode doesn’t pay for a second read path. Fan-in F = 64 counts runs, not inputs: it single-passes any partition of up to 64 budgets of decoded rows while capping worst-case phase-2 residency at (F + 1) × one decoded batch, with each reader’s batch sized from its share of B in bytes (§3.2, #853). Small partitions skip spilling entirely while their decoded rows fit the 64 MiB decoded-row budget; larger ones spill runs of whole inputs as the budget fills (reopened 2026-10-02, §3.2 — originally gated on ≤ 256 MiB of encoded input, which let a partition of many small files decode past a node’s memory limit; issue #895).
  • The H4 wording amendment (§3.3): landed with this slice — docs/hazards.md H4 scopes the 128 MB–1 GB row-group target to ingest-side files and states the compacted threshold as the pruning-granularity knob; the file band and H4.4 file-based detection signals are unchanged.

8. References

  • docs/benchmarks.md §9.13 — the L6 window-browse loss table (runs #8–#17), run #17’s no-bloom-collapse diagnostic, the “write-side layout” lever naming, and the determinism note this RFC’s RFC0036.4 preserves; §9.24 — the authoritative run (k=100 = 1,931,911 B; latency floors 0.370/4.341 pass; the lever “parked on its own line”); §9.7 — D2/D3 at band scale (the 456.7 MiB consolidated file, 166.8 MiB/s).
  • RFC 0031 (comparative program) — §5’s L6 disposition (latency floor gated, storage-bytes published-not-gated) and the frozen- gate set RFC0036.5 must keep green; the #498 scoreboard line this RFC discharges.
  • RFC 0033 (cached template map) — the 187,904 B warm acquisition: the layout-independent floor §2.2 is built on.
  • RFC 0009 (compaction) — the manifest/CAS machinery §3.6 leaves untouched; RFC0009.5 input validation; the D2/D3 measures RFC0036.3 re-asserts. RFC 0022 — promoted attribute columns and the §3.4 re-projection pass clustering rides. RFC 0014 — the flush-policy defaults (ourios-server/src/receiver.rs:55–57) and their §7 knob deferral. RFC 0035 — the ingest concurrency model §3.6 keeps untouched, and RFC0035.5’s intra-partition row-order disclaimer that forces §3.2’s run-formation phase. RFC 0005 §3.5 — the row-group and file bands this RFC re-scopes for compacted output. RFC 0016 — the scanned/pruned counts RFC0036.2 asserts against.
  • Code (paths under crates/ourios-parquet/src/ unless noted): writer.rs:60, writer.rs:627, writer.rs:644 (the 128 MiB uncompressed rotation; no sorting_columns, no max_row_group_size anywhere), crates/ourios-server/src/receiver.rs:55–57 (seal policy), partition.rs:29–43 (PartitionKey — no service dimension), compaction.rs:184–308 (one-file-at-a-time streaming, the §3.2-preserved memory property at compaction.rs:228–234, sorted basenames at compaction.rs:267–268), reader.rs:57 (the streaming ParquetRecordBatchReader §3.2’s merge builds on).
  • docs/hazards.md H4 — the small-file problem: the file band (unchanged), the row-group band (amended, §3.3), and the “sustained … → RFC” escalation this RFC answers. CLAUDE.md §3.5 (no schema change — §3.4 here), §3.6 (local scratch runs are cache, not truth), §2 pillar #1 (pruning via footer reads — the property this RFC restores granularity to).