rfc: 0021 title: Coordinated DataFusion / Arrow upgrade — phased behind upstream status: green author: Jens Holdgaard Pedersen jens@holdgaard.org drafting-assistance: Claude created: 2026-07-03 supersedes: — superseded-by: —

RFC 0021 — Coordinated DataFusion / Arrow upgrade, phased behind upstream

Status: green (2026-08-29) — phase 2a complete; phase 2b still upstream-gated. DataFusion 55.0.0 shipped and fired this RFC’s own reopening trigger, but only half of it: DF 55 carries arrow/parquet 59.2 yet still pins object_store ^0.13.2. Phase 2 is therefore split (§3.2a/§3.2b):

• Phase 2a — landed. arrow 58→59, parquet 58→59, DataFusion 54→55, MSRV 1.88→1.94. thrift leaves the lockfile, clearing GHSA-2f9f-gq7v-9h6m (#295) and the two Dependabot alerts previously dismissed as risk-tolerated; paste leaves too, retiring the RUSTSEC-2024-0436 ignore. RFC0021.8 green; RFC0021.1 re-baselined (below); the deny half of RFC0021.9 green.
• Phase 2b — deferred, trigger unchanged. object_store ≥ 0.14 (RFC0021.7, the CAS re-proof) and the rest of RFC0021.9 (quick-xml ignores, renovate hold #313, #310) open when a released DataFusion accepts object_store 0.14. Tracked by epic #314.

RFC0021.1 was re-baselined, deliberately. It asserted literal versions (arrow 58, datafusion 54.x, MSRV 1.88) and so failed on the phase-2a bump — not because anything regressed, but because a version-literal assertion is a change-detector that fails on every intentional upgrade. Its invariant half (exactly one arrow major, exactly one datafusion) passed throughout, and that is the property the RFC actually bought (it is what forced RFC 0017’s dual decoder when it was violated, #276). The criterion now asserts the invariant plus floors (arrow ≥ 59, DataFusion ≥ 55) — a routine upgrade passes, a downgrade that would drag thrift back fails. Literal versions live where they are executable: Cargo.toml and the lockfile.

(accepted, 2026-08-25, maintainer sign-off: covered the shipped phase 1 scope, RFC0021.1–.6.)

1. Summary

Upgrade the storage/query dependency stack (epic #314) in two phases, each following what upstream has actually shipped:

• Phase 1 (now): DataFusion 53.1 → 54.0, one arrow. DataFusion 54 pins arrow/parquet ^58.3 — the same arrow the querier already pulls — so ourios-parquet moves from arrow/parquet 55.2 to 58.3 and the whole workspace unifies on a single arrow. That removes the RFC 0017 row-path workaround (#276): the dual decoder and the schema_force_view_types = false override exist only because two arrow major versions coexist. MSRV moves 1.85 → 1.88 (DataFusion 54’s floor).
• Phase 2 (when upstream ships it; expected around DataFusion 55): object_store ≥ 0.14 and parquet 59. parquet 59 drops the thrift dependency entirely (clears the GHSA-2f9f-gq7v-9h6m advisory, #295); a DataFusion release that carries object_store 0.14 unblocks #310 and lifts the renovate hold (#313). The quick-xml deny.toml ignores (RUSTSEC-2026-0194/0195) are removed in this phase iff the object_store release pins quick-xml ≥ 0.41 — that may trail phase 2.

The phase boundary is not a preference; it is where upstream currently is: no released DataFusion accepts object_store 0.14 or parquet 59 (DataFusion 54.0.0 pins object_store ^0.13.2, parquet ^58.3.0).

2. Motivation

2.1 The security cluster

thrift 0.17 (GHSA-2f9f-gq7v-9h6m, DoS) enters via parquet 55.2 and is still pinned by parquet 58.3; it is gone only in parquet 59. The advisory is GHSA-only today, so cargo deny does not fail yet — but it will the day RustSec mints an id. The quick-xml DoS advisories (RUSTSEC-2026-0194/0195) are accepted in deny.toml with an explicit removal condition that also sits behind this upgrade chain. Waiting for a single coordinated bump keeps both windows open longer than needed; phase 1 shrinks the eventual security-driven change to a small step.

2.2 The dual-decoder debt (#276)

The RFC 0017 row read path decodes DataFusion’s arrow-58 RecordBatches separately from ourios-parquet’s arrow-55 reader, and pins schema_force_view_types = false to keep the two schemas compatible. Every read-path feature pays this tax twice. Unifying on one arrow removes the second decoder and restores the upstream view-types default.

2.3 Pillar drift

Parquet-on-disk and DataFusion-as-engine are §2 pillars; the longer the stack sits behind upstream, the larger (and riskier) the eventual jump on exactly the code we can least afford to destabilise. Phasing keeps each jump small.

2.4 Why the phases follow upstream

The version locks, as of this writing:

A “single coordinated bump” resolving the whole epic is therefore not constructible from released crates today. What is constructible now — arrow unification — happens to be the riskiest part (it touches the on-disk format pillar), and doing it in isolation means the property/corpus/reconstruction suites validate exactly one change.

3. Proposed design

3.1 Phase 1 — DataFusion 54, arrow 58 everywhere, MSRV 1.88

One coordinated workspace bump: datafusion = 54, and ourios-parquet moves arrow/parquet 55.2 → 58.3 so the lockfile carries a single arrow major. Expected churn:

• ourios-parquet: writer, reader, schema declaration, encode_records_to_parquet — arrow 55 → 58 API changes. This is the load-bearing on-disk format (§2 pillar #1); the §3.3 and §3.5 invariants below bound the change.
• ourios-querier: DataFusion 53 → 54 API changes (logical plans, ListingTable / SessionContext, pruning statistics); removal of the RFC 0017 dual decoder and the schema_force_view_types = false override (#276).
• ourios-bench: compile-level churn only.
• rust-toolchain.toml note + workspace rust-version 1.85 → 1.88 (documented per CLAUDE.md §6.1; CI already runs a newer stable).

What must not change (the §3 invariants this RFC touches):

• On-disk bytes are the contract (§3.5). The upgrade introduces no schema change: field names, types, repetition, and the RFC 0005 §3.2 column set stay identical. Files written before the upgrade MUST read identically after it (RFC0021.2). Any arrow-58 behaviour change that would alter written bytes (encodings, statistics defaults) must be pinned back to the current behaviour or explicitly RFC’d as a §3.5 schema migration — not absorbed silently.
• Bit-identical reconstruction (§3.3). The reconstruction property and corpus tests run unchanged and must stay green.
• The compactor’s conditional-PUT CAS (RFC 0013 §3.3/§3.4) is untouched — object_store does not move in this phase.

3.2a Phase 2a — arrow 59 + parquet 59 (landed 2026-08-28)

DataFusion 55.0.0 pins arrow/parquet ^59.2, so the parquet half of the original phase 2 became available on its own:

• parquet 59 drops thrift → the lockfile has none, cargo deny passes, and the GHSA ignore matches nothing (#295 closed).

• paste also leaves the tree, retiring the RUSTSEC-2024-0436 ignore.

• MSRV 1.88 → 1.94 (DataFusion 55’s floor). That enables clippy’s duration_suboptimal_units, which suggests Duration::from_days / from_mins — both unstable (rust-lang/rust#120301) — so it is allowed workspace-wide with that reason rather than chased into nightly-only APIs.

• API churn was two items: an arrow-cast pin, and DataFusion 55’s new required ExecutionPlan::apply_expressions (three test doubles in ourios-df-otel). No production code changed.

• execution.collect_statistics is turned off on the querier’s session (ourios_querier::exec::session), which is the one behavioural decision in this phase. DataFusion 55 substitutes columns that per-file statistics prove constant; the all-NULL case folds to a NULL literal, collapsing predicates like body == "…" to a bare constant, which leaves no column reference for a pruning predicate — so RFC 0044’s body-equality queries scan every row group instead of skipping them (pillar #1, and the RFC0044.7/.8 + RFC0007.1 tests caught it). Upstream bug [apache/datafusion#24769]; fix proposed in [apache/datafusion#24770], bisected to DataFusion #22969 which made collect_statistics the session-level default.

  The override is not purely defensive: statistics collection also costs a footer read per file at plan time, which a many-file log store pays on every query, and Ourios derives its own pruning from the RFC 0009 manifest plus partition-directory windowing rather than from DataFusion’s collected file statistics. Revisit when the upstream fix ships in a release we depend on — the same three pruning tests are the gate in both directions.

3.2b Phase 2b — object_store ≥ 0.14 (still upstream-gated)

DataFusion 55 still pins object_store ^0.13.2, so this half did not open. Trigger unchanged: a released DataFusion that accepts object_store 0.14. Scope, known today:

• object_store 0.13 → 0.14+ API churn concentrated in crates/ourios-parquet/src/store.rs (AmazonS3Builder, PutMode/PutOptions, S3ConditionalPut, list/ list_with_delimiter, UpdateVersion). The compactor’s publish-CAS (RFC0013.3/.4) must be preserved and is re-proven by the existing localstack suite.
• parquet 59: thrift leaves the lockfile → close #295.
• Supply chain: object_store 0.14 pulls new transitives (aws-lc-rs / aws-lc-sys family among them) — deny.toml licenses/advisories re-cleared, osv-scanner.toml updated if needed.
• Renovate: lift the <0.14.0 hold (#313); close #310.
• quick-xml: drop the RUSTSEC-2026-0194/0195 ignores iff the object_store release pins quick-xml ≥ 0.41; otherwise the ignores stay with their documented removal condition.

3.3 Non-goals

No Parquet schema change, no logs-DSL surface change, no Store trait change, no query-semantics change. This RFC is a dependency migration with pinned invariants, not a feature vehicle.

4. Alternatives considered

4.1 Wait for DataFusion 55 and do one coordinated bump

Rejected: it couples the riskiest migration (arrow 55 → 58 on the on-disk pillar) with the object_store API churn in the same change, leaves #276’s dual decoder and the security windows open for longer, and gambles on DataFusion 55’s actual contents. Phase 1 is exactly the de-risking slice: upstream’s own arrow unification with nothing else moving.

4.2 Fork/patch object_store 0.13 onto quick-xml 0.41

Rejected: a patched fork of a supply-chain-sensitive crate trades a documented, low-exposure DoS ignore for permanent maintenance burden and a worse provenance story.

4.3 Bump only ourios-parquet to parquet 59 (thrift fix first)

Rejected: parquet 59 means arrow 59 in ourios-parquet while DataFusion 54 carries arrow 58 — reintroducing the dual-arrow split (#276) one version higher, on the read and write path this time.

5. Acceptance criteria

Scenario ids RFC0021.<m>. Phase 1 = .1–.6; phase 2a = .8 and the deny half of .9. This section lists only criteria whose trigger has fired — green means all of them pass, per docs/rfcs/README.md. Phase 2b’s criteria live in §5b below and are inactive: they enter this ledger (as red stubs, moving the RFC back to red) the day a released DataFusion accepts object_store ≥ 0.14, and not before.

Status note (2026-08-29): phase 1 discharged 2026-07-03 (#339, #340; indicative B1/B2 ci-runner run on #340). Phase 2a discharged 2026-08-29 (.1 re-baselined per the header, .8 landed, .9’s deny half green). The 2026-07-03 note’s “opening phase 2 moves the status back to red” now applies to phase 2b — 2a opened and closed within one PR, so the red window collapsed to nothing.

Scenario RFC0021.1 — one arrow (re-baselined, see header). Given the current dependency set, When the workspace lockfile is inspected, Then exactly one arrow major and exactly one datafusion version are present, with floors arrow ≥ 59 / DataFusion ≥ 55 (a downgrade that would drag thrift back fails), and the workspace builds with the pinned MSRV (1.94).

Scenario RFC0021.2 — old files read identically (§3.5). Given Parquet files written by the pre-upgrade writer (committed fixture + freshly generated), When the post-upgrade reader reads them, Then every row and column decodes identically to the pre-upgrade reader’s output, with no schema-mismatch errors.

Scenario RFC0021.3 — reconstruction stays bit-identical (§3.3). Given the existing reconstruction property and corpus suites, When they run on the upgraded stack, Then they pass unchanged (no test weakened or deleted).

Scenario RFC0021.4 — the dual decoder is gone (#276). Given the RFC 0017 row read path, When a query renders rows end-to-end, Then decoding goes through the single unified arrow path, schema_force_view_types is no longer overridden, and the RFC 0017 suites pass.

Scenario RFC0021.5 — the pruning thesis holds (B1/B2). Given the benchmarks.md B1/B2 gates, When the query benchmarks run on the upgraded stack (indicative ci-runner pass; authoritative baseline rerun on maintainer opt-in), Then selective-query row-group pruning shows no regression beyond run-to-run noise.

Scenario RFC0021.6 — the full gate is green. Given the phase-1 change, When CI runs, Then the complete suite passes — including s3 integration (localstack) (the CAS paths, untouched) and live-check (weaver).

Scenario RFC0021.8 (phase 2a — green) — thrift is gone. Given the parquet 59 bump, When the lockfile is inspected, Then no thrift crate is present (#295 closed). Asserted by rfc0021_8_thrift_is_absent_from_the_lockfile.

Scenario RFC0021.9 (phase 2a half) — supply chain re-cleared. Given the new transitive set, When cargo deny check runs, Then it passes (it does, and the dead RUSTSEC-2024-0436 paste ignore was removed). The quick-xml half of this criterion is phase 2b’s — see §5b.

5b. Phase 2b criteria (inactive — trigger not fired)

These activate, as red stubs in §5, when a released DataFusion accepts object_store ≥ 0.14 (epic #314). Until then they are forward declarations, not acceptance criteria, and do not count against the status.

Scenario RFC0021.7 — CAS survives object_store 0.14. Given the object_store bump, When the RFC0013.3/.4 conditional-PUT localstack suites run, Then concurrent-sweep publish semantics are preserved.

Scenario RFC0021.9 (phase 2b half) — quick-xml ignores retired. Given the object_store ≥ 0.14 transitive set, When cargo deny check runs with the renovate hold lifted (#313), Then it passes with the quick-xml ignores removed iff object_store pins quick-xml ≥ 0.41.

6. Testing strategy

Per CLAUDE.md §6.2. The existing suites are the oracle — this RFC adds one artefact and changes no test semantics:

• Fixture for RFC0021.2: before the bump, a small Parquet file (representative rows: structured + templated bodies, attributes, non-finite doubles) is generated by the current writer and committed under testdata/; a new test reads it and asserts decoded equality against its committed expected rows. This makes “old files still read” a permanent regression test, not a one-off migration check.
• Property/corpus/reconstruction suites (§3.3), the RFC 0005 §3.9 absent-column tests, and the RFC 0017 suites run unchanged.
• RFC0013.3/.4 localstack CAS tests re-prove the store seam (phase 1: unchanged; phase 2: the actual subject).
• Benchmarks: B1/B2 indicative on ci-runner first, per the standing bench policy; the paid baseline rerun only on explicit opt-in.

7. Open questions

• DataFusion 55 contents and timing — does it pick up object_store 0.14 and parquet 59 together? (Determines whether phase 2 is one step or two.)
• aws-lc-rs / aws-lc-sys license family under deny.toml’s allow-list when object_store 0.14 lands (ISC + Apache-2.0 variants; needs review, possibly an exceptions entry).
• Is an authoritative baseline B1/B2 rerun wanted after phase 1, or indicative-only until phase 2 completes the epic?
• MSRV cadence: 1.85 → 1.88 is forced here; do we want a documented policy (e.g. “MSRV may follow DataFusion’s floor”) instead of per-RFC decisions?

8. References

• Epic #314 (this RFC), #310 (object_store 0.14, blocked), #295 (thrift GHSA-2f9f-gq7v-9h6m), #276 (RFC 0017 dual decoder), #313 (renovate hold).
• RFC 0011 (A1 demotion — context for the bench gates), RFC 0013 (Store / object_store seam, CAS), RFC 0017 (row read path).
• CLAUDE.md §2 (pillars #1, #3), §3.3 (bit-identical reconstruction), §3.5 (schema migration), §6.1 (MSRV), §6.2 (tests are specifications).
• deny.toml RUSTSEC-2026-0194/0195 ignore block (removal condition).