rfc: 0001 title: Template miner (Drain-derived online log parsing) status: accepted author: Jens Holdgaard Pedersen jens@holdgaard.org drafting-assistance: Claude created: 2026-04-24 supersedes: — superseded-by: —

RFC 0001 — Template miner

Status note. accepted (2026-06-14, maintainer sign-off — the terminal ladder status per docs/rfcs/README.md). Reached validated the same day on the evidence below; accepted records the maintainer’s final sign-off on the template-mining pillar. The docs/verification.md §3 / docs/rfcs/README.md ladder reserves validated for every thesis-gate the RFC’s pillars touch passing on representative corpora (benchmarks.md §7). The template-mining pillar’s gates are C1 (reconstruction fidelity) and C2 (template-count convergence) — both pass on the representative LogHub HDFS_v1 corpus (≈ 1.47 GiB, 11.2 M lines — above benchmarks.md §8’s ≥ 1 GiB canonical floor, so representative; and well past C2’s ≥ 1 M-line formal-gate threshold, so that gate applies rather than abstains), authoritatively on the benchmarks.md §1 baseline hardware: C1 1.000000, C2 a 40-template plateau (diagnostic local run benchmarks.md §9.5, authoritative baseline-8vcpu-32gib rerun §9.6 — identical verdicts, as expected of deterministic gates). A1 (compression vs zstd) is a diagnostic, not a gate (RFC 0011, already encoded in the §7 gate table): it fails on every corpus class including the maximally-templated one, for structural reasons — template mining’s compression is logical/query-pruning (B1/B2), not on-disk bytes.

Reached green first (2026-06-13): all §5 acceptance criteria pass with live tests (zero #[ignore]/todo!() stubs) — miner-internal (tokenize/mask/sim-seq, the three-zone confidence model, fresh-leaf + widening + type-expansion with audit events, the (severity_number, scope_name) template key, the 256 B param-overflow spill + telemetry, bit-identical reconstruction + the H7.3 render contract, structured-body canonical encoding, the §6.9 snapshot + v2 restore) plus the relocated cross-crate criteria (query semantics RFC0001.5/.6, time-preserved RFC0001.10, §3.7.3 per-ResourceLogs tenant derivation, drift H5.3 via RFC 0010, the §6.7 alias index via RFC 0005 §3.7).

Terminal step: accepted is the maintainer’s final sign-off (docs/rfcs/README.md). NB the A1 re-scope’s own RFC (RFC 0011) is still drafted; the §7 gate table already reflects the demotion, but RFC 0011 should be accepted to fully ratify that chain.

How to read this document. §§1–4 are the design contract — the what and the why. §5 lists the normative Given / When / Then scenarios — the contract — grouped by parent (hazard, invariant, RFC-internal). §6 is the precise specification the ourios-miner crate is implemented against; its opening paragraphs name the gaps between the published algorithm and a production miner that §6.1–§6.9 then close. §7 records the alternatives we evaluated and rejected. §8 maps each §5 scenario to the technique that tests it.

Cross-references to CLAUDE.md sections are in square brackets, e.g. [§3.1], and name the invariant the section must preserve.

1. Summary

Ourios implements a Drain-derived online template miner (ourios-miner) that converts each ingested OTLP LogRecord into a structured Parquet record. The record shape is the OTLP LogRecord (with its inherited Resource and InstrumentationScope context) plus the miner-derived columns (template_id, template_version, params, separators, body_kind, body?, confidence, lossy_flag); see §6.1 for the full schema and the 2026-05-13 amendment that aligned it to OTLP. The miner is per-tenant by construction [§3.7], uses a three-zone confidence model that retains the original line in the lossy zone [§3.1], audits every template widening [§3.1], captures inter-token separators in a parallel array so that bit-identical reconstruction is the default rather than a property-test exception [§3.3], bounds parameter values at 256 B with overflow to a side body column [§3.2], and tracks template structural changes via a monotonic template_version so that schema drift across deploys is a first-class query rather than a silent count drop [§3.5]. The 50–200× figure is a logical reduction (lines → (template_id, params)), realised as query pruning (gates B1/B2), not on-disk bytes versus a byte codec — see RFC 0011, which demoted the compression-vs-zstd ratio (A1) to a diagnostic.

2. Motivation

This is the load-bearing pillar of the project [§2.2]. Three sub-questions justify it.

Why template mining at all. A typical service emits 10²–10⁴ distinct printf templates over its entire lifetime, but raw log volume is dominated by the parameters substituted into those templates. Storing the template once per tenant and the parameters per occurrence makes that redundancy explicit — and explicit redundancy stacks with byte-level codecs rather than fighting them. zstd over flat log text recovers ~10× on typical workloads; doing the structural work first leaves zstd a column of short, repetitive parameters that dictionary-encode well, where the codec then earns its keep again. The 50–200× headline (README.md, [§2.2]) is the product of these two layers, not a claim about either alone.

Why online vs. offline. Operators expect logs to be queryable within seconds of ingest, not minutes. Any batch clustering window long enough to do offline hierarchical clustering well is a window the operator is blind in. Drain’s fixed-depth tree gives O(d) lookup per line at the cost of being slightly less accurate than the best offline parsers — an acceptable trade because §3.1’s audit and confidence machinery surfaces the inaccuracy rather than hiding it.

Why this layer. The compression is structural, not statistical. Doing it before Parquet’s byte codecs means each Parquet column sees small, dictionary-friendly values; doing it after means we have already paid for storing the redundancy and zstd has to find it again from the bytes. The order matters.

3. Background: Drain as published

A restatement of He et al., ICWS 2017, in the notation this RFC uses downstream. Citations are by paper section/figure.

3.1 Tree structure (paper §3.2, Fig. 2)

A fixed-depth parse tree, depth d (default 4 in the paper). Three node kinds, in order from root:

• Root. Single node; routes by token count.
• Length-N node. One per observed token count N. Children are prefix nodes keyed by the first token of the line.
• Token-prefix nodes at depths 2..=d. Each is keyed by the token at position (depth - 1) of the line.
• Leaf log groups at depth d + 1. Each leaf holds a template — a sequence of N tokens, where each position is either a fixed string or the wildcard <*>.

                   root
                    │
        ┌───────────┼───────────┐
       len=4       len=5       len=6      ← length groups
        │           │           │
    ┌───┴───┐    ┌──┴──┐     ┌──┴──┐      ← prefix nodes (depth 1)
   "user"   …  "GET"  …    "INFO"  …
    │           │           │
   ┌┴┐         ┌┴┐         ┌┴┐
   …  …        …  …        …  …          ← prefix nodes (depth 2)
   │           │           │
 [leaf]      [leaf]      [leaf]           ← log groups

3.2 Similarity function (paper §3.3)

For a candidate line L = (t_1, …, t_N) and a leaf template T = (τ_1, …, τ_N):

simSeq(L, T) = (count of positions i where t_i == τ_i or τ_i is <*>) / N

Wildcards in the template count as matches. The line length and the template length are equal by construction (the length-N node selected the leaf candidates).

3.3 Threshold st (paper §3.4)

A configured value st ∈ (0, 1]. After computing simSeq against every leaf at the current parent, the leaf with the highest simSeq is the candidate. If simSeq(L, T_best) ≥ st, the line attaches to T_best; otherwise a new leaf is created. The paper reports st = 0.4 as a default; see §6.3 for why Ourios overrides this.

3.4 New-log-group creation vs. leaf update (paper §3.5)

If simSeq(L, T_best) < st, a new leaf is created at the parent prefix node, with L as its initial template (no wildcards yet). Otherwise T_best is updated: at every position where t_i ≠ τ_i, the template position is replaced with <*>. The template never becomes more specific over time, only more general; positions can become wildcards but cannot become fixed again.

3.5 Worked example

A fabricated illustration (no testdata/corpus/ exists yet; this example will be replaced with one drawn from the corpus once it lands).

Line A:  user 42 logged in from 10.0.0.1
Line B:  user 17 logged in from 10.0.0.2
Line C:  user 99 logged out from 10.0.0.7

After preprocessing (§4.2), numbers and IPs are masked, so the miner sees:

Line A:  user <NUM> logged in from <IP>
Line B:  user <NUM> logged in from <IP>
Line C:  user <NUM> logged out from <IP>

All three are length 6. They route to root → len=6 → "user".

A walks the prefix path further (depth 2: token at position 1 is the masked <NUM> placeholder, treated as a fixed token at this stage). It is the first line, so a leaf is created with template user <NUM> logged in from <IP>.

B walks the same path. The candidate leaf has simSeq(B, T_A) = 6/6 = 1.0 ≥ st. B attaches; the template is unchanged.

C walks the same path. The candidate leaf has simSeq(C, T_A) = 5/6 ≈ 0.833. With st = 0.7 (Ourios default, §6.3), 0.833 ≥ 0.7, so C attaches. Token position 4 (in vs out, 1-indexed against the masked sequence) becomes <*>. The template widens to user <NUM> logged <*> from <IP>. This is a template widening event and must emit an audit record per §6.4.

4. Background: Drain3 extensions (not in the paper)

Drain3 (logpai/Drain3) is the maintained Python implementation. It adds several capabilities beyond the 2017 paper. Each is recorded here as adopt, adopt with modification, or reject, with one sentence of rationale.

4.1 Persistent state — adopt with modification

Drain3 supports JSON snapshots to file, Redis, or Kafka. Ourios adopts the snapshot concept but commits to a file/object-storage backend; Redis and Kafka are out of scope (CLAUDE.md §3.6 names object storage as the source of truth). Snapshot target, cadence, and scope are open questions in §9.

4.2 Pre-tree-walk masking — adopt with modification

Drain3’s most important extension: regex-based masking of common parameter shapes (IPs, UUIDs, numbers, hex, timestamps, file paths) before the tree walk, so high-cardinality tokens never become tree branches. Without this, the tree explodes into one branch per IP address.

The Ourios modification: a masked token is not discarded. It becomes a typed parameter attached to the wildcard slot it created. The masking layer emits (type_tag, original_bytes) pairs; the tree walk treats the type tag as the token (so <NUM> matches <NUM> for tree-routing purposes) while the original_bytes flow into params so reconstruction can recover the line exactly. Paper-pure Drain loses the original token; Ourios retains it as a parameter. This is what makes [§3.3] reconstruction possible at all.

4.3 Variable-length wildcards — adopt with constraint

Drain3’s MaskingInstruction allows a single regex to match a variable-length run of tokens (e.g. a multi-token user-agent string). Ourios adopts this where the run is bounded at parse time and produces exactly one typed parameter in the output. Reject: unbounded variable-length wildcards, because they break leaf identity (two lines with the same template structure but different run lengths would land in different length-N nodes and never deduplicate).

4.4 Dynamic / adaptive threshold — reject

Drain3 supports auto-tuning the similarity threshold per leaf based on observed cluster sizes. Ourios rejects this. CLAUDE.md §3.1 fixes threshold ≥ 0.7 as a project-level invariant; auto-tuning would silently move the merge boundary across deploys, defeating the audit contract. Threshold tuning is a config decision per tenant, never a runtime decision per leaf.

4.5 Other Drain3 features

• Parameter-naming hints. Drain3 lets users name <*> slots via the masking config (e.g. <IP:client_addr>). adopt — the type-tag mechanism in §4.2 already requires a slot name; using the Drain3 hint format keeps configs portable.
• Built-in metrics surface. Drain3 exposes a set of state counters via callback. replace — Ourios exposes OTel metrics directly per §6.8 (instrumented via the meter API), with names that match [§3.1]’s required set rather than Drain3’s internal names.
• Parameter masking after the fact. Drain3 has utilities to retroactively mask params in already-clustered lines. reject — Ourios masks once, at ingest, deterministically. Retroactive masking would invalidate already-written Parquet files.

5. Acceptance criteria

Per docs/verification.md §§2–3, every CLAUDE.md §3 invariant and every docs/hazards.md hazard this RFC touches has at least one numbered scenario below. Scenarios use the bold-leading-clause format (verification.md §2.1) and the id grammars (§2.2): H<n>.<m> for hazard-rooted, §3.<n>.<m> for invariant-rooted, RFC0001.<m> for design-internal commitments. Test code carries each id in a doc comment per §2.3 so grep -R "H1.1" . resolves bidirectionally between RFC and tests.

The hazards in scope are H1, H2, H5, H7; the invariants are §3.1, §3.2, §3.3, §3.5, §3.7. H3 (WAL durability) and H4 (small files) are owned by the ourios-wal and ourios-parquet RFCs; H6 (DSL) is owned by RFC 0002; §3.4 (WAL-before-ack) and §3.6 (object-storage-as-truth) are touched only via §6.9’s persistence direction and the primary obligation lives in those other RFCs.

5.1 Hazards

Scenario H1.1 — Semantically distinct templates do not silently merge

• Given a corpus containing user logged in <*> and user logged out <*>
• When similarity threshold is 0.7 (the default)
• Then the two remain distinct template_ids
• And any widening produces an audit event recording both old and new templates

Scenario H1.2 — Lossy-zone match retains body

• Given a line whose best match has confidence in the lossy zone (floor ≤ x < threshold)
• When the line is ingested
• Then the body column contains the original line bytes
• And the row carries lossy_flag = false (the flag is reserved for tokenizer / preprocessing failure per §6.6 — the lossy zone retains the body but reconstruction still succeeds)

Scenario H1.3 — Every widening emits an audit event

• Given any sequence of inputs that triggers a template widening
• When the widening completes
• Then an audit event exists naming the old template, the new template, the tenant id, the timestamp, and the event_type

Scenario H1.4 — severity_number is part of the template key (no INFO/ERROR silent merge)

• Given two OtlpLogRecords with identical body_kind = String bodies and identical scope_name, but severity_number = 9 (INFO) and severity_number = 17 (ERROR)
• When both are ingested via MinerCluster::ingest
• Then the emitted records carry distinct template_ids
• And no widening or merge ever produces a single template_id covering both severity buckets
• (Operationalises the §6.1 Template-key composition commitment that severity_number is part of the key regardless of body_kind.)

Scenario H1.5 — scope_name is part of the template key (no cross-scope silent merge)

• Given two OtlpLogRecords with identical body_kind = String bodies and identical severity_number, but scope_name = Some("lib.auth") and scope_name = Some("lib.payments")
• When both are ingested
• Then the emitted records carry distinct template_ids
• And no widening or merge ever produces a single template_id covering both scopes
• And a third record with scope_name = None shares a template_id with neither (it lives in the (severity, None) bucket per §6.1)

Scenario H2.1 — Oversized parameter triggers OVERFLOW marker and forced body retention

• Given a tenant configured with the default 256 B per-parameter byte limit
• And a log line whose masked parameter value exceeds 256 B (e.g. an embedded stack trace)
• When the line is ingested
• Then the corresponding Param entry has type_tag = OVERFLOW carrying (length, sha256_prefix) instead of the original value
• And the body column contains the original line bytes regardless of lossy_flag
• And ourios.miner.params.overflow (attributes ourios.tenant, ourios.service) increments

Scenario H2.2 — Per-service overflow rate above 1% raises an alert

• Given the ourios.miner.params.overflow.utilization gauge (attributes ourios.tenant, ourios.service) for some service
• When the rolling rate exceeds 0.01
• Then the documented alert rule fires (the rule ships alongside §6.5’s metric definition)

Scenario H5.1 — Wildcard widening increments template_version and emits template_widened

• Given a leaf at (template_id = X, template_version = V)
• When an attach widens a previously-fixed token at position i into <*>
• Then the leaf’s template_version becomes V + 1
• And an audit event with event_type = template_widened is emitted naming the new wildcard position(s)

Scenario H5.2 — Type expansion increments template_version and emits template_type_expanded

• Given a leaf whose wildcard slot s has slot_types[s] = {NUM}
• When an attach maps a typed parameter of type = STR into slot s
• Then slot_types[s] becomes {NUM, STR}
• And template_version increments
• And an audit event with event_type = template_type_expanded is emitted naming the slot and the newly-added ParamType

Scenario H5.3 — Drift query returns templates that gained a version in window

• Given the template_audit event stream contains template_widened and template_type_expanded events for templates A and B in the window [t1, t2]
• When the §6.7 drift query runs against [t1, t2]
• Then the result includes both A and B with their widening counts

Scenario H7.1 — Reconstruction property holds across the corpus

• Given the committed testdata/corpus/ (anonymised, fixed)
• When every line is ingested through the miner
• Then for every emitted record r where r.lossy_flag = false, reconstruct(r) == r.ingested_bytes holds byte-for-byte
• And property failure is a build break, not a regression

Scenario H7.2 — Tokenizer failure sets lossy_flag = true and retains body

• Given a line containing an embedded NUL byte (or another tokenizer-failure mode listed in §6.6)
• When the line is ingested
• Then a parse-failure record is emitted
• And the record’s lossy_flag is true
• And the record’s body column contains the original line bytes

Scenario H7.3 — Reader emits body verbatim when lossy_flag is true

• Given a record with lossy_flag = true
• When the reader renders the row
• Then the rendered bytes are the body column verbatim (byte-for-byte, no prefix or in-band marker)
• And the rendered row carries the §6.6 reconstruction signal Reconstruction::RetainedVerbatim (the out-of-band warning marker the §6.6 Reader render contract defines)
• And reconstruct() is NOT called for that row

Scenario H7.4 — Widened literal slot reconstructs via STR fallback

• Given a leaf whose template gains a new <*> slot at position i via the §6.2 widening of an originally-literal token
• When the triggering line is attached
• Then the line’s record carries params[slot_for_i] = { type_tag: STR, value: L_tok[i] }
• And reconstruct(record) == ingested_bytes holds

5.2 Invariants

Scenario §3.1.1 — Default similarity threshold is 0.7

• Given a tenant configuration with no threshold override
• When the miner is initialised for that tenant
• Then the effective threshold is 0.7

Scenario §3.1.2 — Mandatory metric set is exposed

• Given the mandatory set defined by §6.8’s table — the ourios.miner.* metric instruments in the semconv registry (semconv/registry/, surfaced as generated ourios_semconv constants) that the miner registers on the ourios.miner meter when it is constructed (the ourios.miner.alias.* counters are registered separately by the alias map, §6.7, and are out of this scenario’s scope)
• When a small representative workload exercises every instrument (a normal line, a near-duplicate that widens a template, an oversized-param line, and a parse-failure line) and the meter is collected via an SDK in-memory reader
• Then the collected metric stream contains every metric named in §6.8’s table — each appearing on its first real measurement, carrying the registry’s required attributes (no synthetic zero-traffic points) — (ourios.miner.template.count, ourios.miner.merges, ourios.miner.confidence, ourios.miner.confidence.p50, ourios.miner.confidence.p01, ourios.miner.body_retention.utilization, ourios.miner.parse_failures, ourios.miner.params.overflow, ourios.miner.params.overflow.utilization, ourios.miner.template.version_changes, ourios.miner.duration) with the instrument kinds and attributes listed there

Scenario §3.2.1 — Default per-parameter byte limit is 256

• Given a tenant configuration with no per-parameter byte limit override
• When the miner is initialised for that tenant
• Then the effective limit is 256 bytes

Scenario §3.2.2 — Configured limit above 1 KiB is rejected at startup

• Given a tenant configuration with param_byte_limit > 1024
• When the miner is initialised
• Then initialisation fails with an error citing the §3.2 ceiling
• And the process refuses to start serving that tenant

Scenario §3.3.1 — Separators array captured on every successful tokenization

• Given a line that tokenizes successfully
• When the line is ingested
• Then the emitted record’s separators.len() == tokens.len() + 1
• And the per-row precondition for H7.1 holds (the reconstruction proptest then asserts byte equality)

Scenario §3.5.1 — Snapshot format carries a leading version byte

• Given a serialised snapshot artefact written by the miner
• When the artefact is inspected
• Then byte 0 is the snapshot format version

Scenario §3.5.2 — Unknown snapshot version triggers full WAL replay

• Given a snapshot artefact whose leading version byte is unknown to the running miner
• When the miner loads the snapshot at startup
• Then the snapshot is rejected
• And the miner falls back to full WAL replay rather than misinterpreting the bytes

Scenario §3.5.3 — Known-version restore + tail replay is equivalent to a full rebuild (2026-06-12 amendment)

• Given a tenant tree snapshotted at WAL high-water mark S, with further frames appended after S
• When recovery restores the snapshot and replays only the frames above S
• Then the recovered tree state (leaves, template_ids, template_versions, slot types, structured-template-id map) equals the control tree built by ingesting every record from scratch
• And no frame at or below S reaches the miner (no double-apply — the v1 hazard that gated restore)

Scenario §3.5.4 — Stale snapshot degrades loudly, not silently (2026-06-12 amendment)

• Given a snapshot at high-water mark S, a Parquet checkpoint at X > S, and a WAL whose surviving segments start above S but retain every frame above X (externally truncated — WAL segment files manually unlinked; the RFC 0008 §6.7 retain floor prevents this arising internally, and legitimate housekeeping never removes a frame above X)
• When recovery runs
• Then the snapshot is restored and the surviving frames are replayed under the per-consumer horizons (the data side is complete: every missing frame was ≤ X, hence already in Parquet)
• And a structured warning is emitted naming the gap between S and the oldest surviving frame, so the possible template re-minting inside it is surfaced (hazard #5, observable via the RFC 0010 drift query) rather than silent

Scenario §3.7.1 — Tenants’ template trees never cross-pollinate

• Given a MinerCluster ingesting interleaved lines from synthetic tenants A and B
• When the corpus is fully ingested
• Then no template mined under tenant A appears in tenant B’s tree
• And no template mined under tenant B appears in tenant A’s tree
• (Implements docs/benchmarks.md E2.)

Scenario §3.7.2 — Same structural template in two tenants gets distinct template_ids

• Given tenants A and B independently emit the structurally identical template user <NUM> logged in from <IP>
• When both are ingested
• Then tenant A’s template_id for that template differs from tenant B’s template_id
• And no template_id is shared across tenants
• And template_ids are guaranteed unique across the entire cluster (not just per tenant)

Scenario §3.7.3 — Tenant derivation runs per ResourceLogs, not per export batch

• Given a single OTLP ExportLogsServiceRequest carrying two ResourceLogs whose Resource.attributes resolve to distinct tenants A and B under the configured derivation rule
• When the receiver fans the batch out per RFC 0003 §6.3 and the miner ingests both per-tenant streams
• Then every LogRecord under ResourceLogs[0] is mined under tenant A
• And every LogRecord under ResourceLogs[1] is mined under tenant B
• And no record ever appears in the wrong tenant’s tree
• (Operationalises the §6.1 Tenant derivation commitment that the derivation rule runs once per inherited Resource, not once per export batch.)

5.3 RFC-internal design commitments

Scenario RFC0001.1 — Fresh-leaf creation does not emit an audit event

• Given a parent prefix node with no leaves yet
• When a line creates the first leaf at that node
• Then no event is appended to the audit stream for that creation
• And ourios.miner.template.count increments to reflect the new leaf

Scenario RFC0001.2 — Degenerate-template guard rejects fully-wildcard widening

• Given a leaf whose template, after a candidate widening, would have zero non-wildcard tokens
• When the candidate widening is attempted
• Then the widening is rejected
• And the line is treated as a parse failure (confidence = 0, body retained, ourios.miner.parse_failures increments)
• And an audit event with event_type = template_widening_rejected_degenerate records the rejection

Scenario RFC0001.3 — Tokenizer is Unicode whitespace only; punctuation stays in tokens

• Given a line key=value, other=42 (no whitespace adjacent to the punctuation)
• When the line is tokenized
• Then it produces two tokens (key=value, and other=42)
• And no token boundary is introduced at =, ,, :, ;, [, ], (, or )

Scenario RFC0001.4 — Confidence ratio = simSeq / threshold; decision boundary at 1.0

• Given a tenant with threshold = 0.7
• And a line whose simSeq against the best candidate is 0.7
• When the line is ingested
• Then the emitted record’s confidence == 1.0
• And the line takes the clean-attach branch
• And the same simSeq under threshold = 0.5 would yield confidence == 1.4 (the ratio reframes scale-invariantly across tenants)

Scenario RFC0001.5 — Bare template_id = X spans all versions of leaf X

• Given leaf X with versions 1, 2, 3 attached over time
• When a query runs where template_id = X
• Then the result includes rows attached against (X, 1), (X, 2), and (X, 3)
• And no alias resolution is involved (this is by-construction, since template_id is stable across widenings of one leaf)

Scenario RFC0001.6 — Bare template_id = X does NOT follow alias chains

• Given two distinct leaves X and Y that the alias index records as semantically equivalent
• When a query runs where template_id = X
• Then only rows whose template_id == X are returned; rows with template_id == Y are NOT included
• And where template_id.resolves_to(X) (RFC 0002 §5.4) is the explicit form that includes Y’s rows

Scenario RFC0001.7 — Combined widening + type-expansion increments version twice and emits two events in order

• Given a leaf at version V where a single attach both introduces a new wildcard slot AND introduces a previously-unseen ParamType into an existing slot
• When the attach completes
• Then the leaf’s template_version == V + 2
• And the audit stream contains two events for this attach: a template_widened event for the new wildcard, immediately followed by a template_type_expanded event for the type expansion (in that order)

Scenario RFC0001.8 — ourios.miner.confidence.p50 and ourios.miner.confidence.p01 are emitted as gauges

• Given a running miner with a non-empty ourios.miner.confidence histogram for some (ourios.tenant, ourios.service)
• When the miner’s meter is collected via an SDK in-memory reader
• Then ourios.miner.confidence.p50 and ourios.miner.confidence.p01 (attributes ourios.tenant, ourios.service) are present as gauges
• And each value matches the corresponding quantile of the same-attributed histogram (computed in-process on a short ticker per §6.8)

(The dotted-semconv rename landed in the 2026-06-08 amendment; the open fork is whether confidence.p50 / confidence.p01 become backend-derived quantiles over the exported histogram rather than in-process gauges. That is a contract change to the §3.1.2 mandatory set and would be made — possibly superseding this scenario — under its own review, not folded into the rename.)

Scenario RFC0001.9 — body_kind = Structured short-circuits to a structured-template id

• Given an OtlpLogRecord whose body is Body::Structured(AnyValue) (any non-String AnyValue variant carried verbatim per RFC 0003 §6.4)
• When the record is ingested
• Then the §6.2 algorithm skips tokenize/mask/descend per step 0 and allocates or reuses the structured-template id for (severity_number, scope_name, BodyKind::Structured)
• And the emitted record has body_kind = Structured
• And the emitted record’s body carries the Ourios canonical body encoding of that AnyValue (per the §6.1 encoding rule)
• And params and separators are empty
• And confidence == 1.0 (the §6.1 sentinel)
• And lossy_flag == false

Scenario RFC0001.10 — time_unix_nano is preserved verbatim from the wire

• Given an OtlpLogRecord with time_unix_nano = 1_715_700_000_000_000_000
• When the record is ingested and committed to Parquet
• Then the emitted row has time_unix_nano == 1_715_700_000_000_000_000
• And a query WHERE time_unix_nano BETWEEN 1_715_600_000_000_000_000 AND 1_715_800_000_000_000_000 returns the row
• (Gates docs/benchmarks.md B1 — time-range queries — by making the underlying column measurable.)

Scenario RFC0001.11 — severity_number = 0 and scope_name = None are distinct key buckets

• Given four OtlpLogRecords with identical body_kind = String body, varying only in (severity_number, scope_name) across (0, None), (0, Some("lib.x")), (9, None), (9, Some("lib.x"))
• When all four are ingested
• Then four distinct template_ids are emitted, one per key bucket
• And no widening or merge ever coalesces the severity_number = 0 (UNSPECIFIED) bucket with any specified-severity bucket
• And no widening or merge ever coalesces the scope_name = None bucket with any scope_name = Some(_) bucket
• (Locks the §6.1 explicit edge-case rules: 0 = UNSPECIFIED is a valid OTLP severity that gets its own bucket, and absent scope is its own bucket.)

Scenario RFC0001.12 — Alias assertion is durably recorded and appears in the per-tenant map

• Given tenant T with two distinct leaves A and B (A < B) and no existing alias set
• When an operator asserts B is an alias of A
• Then an alias_asserted audit event is durably recorded on the §6.4 stream under the §3.4 WAL-before-ack barrier before the assertion is acknowledged, naming tenant_id = T, the anchor representative_id = A, member_ids = [B], the actor, and the timestamp — so the asserted set is {A} ∪ {B} = {A, B}
• And after the per-tenant projection rebuilds, tenant T’s alias map contains an equivalence class with members {A, B} whose derived canonical representative is A (the smallest member)

Scenario RFC0001.13 — resolves_to(rep) returns all members and excludes non-members

• Given tenant T whose alias map records the set {A, B} (per RFC0001.12) and an unrelated leaf C in no set
• When the querier compiles template_id.resolves_to(A) for tenant T
• Then the predicate expands to template_id IN {A, B}
• And resolves_to(B) expands to the same {A, B} (expansion is by the set, not the direction of assertion)
• And resolves_to(C) expands to exactly {C}

Scenario RFC0001.14 — Cross-tenant isolation: an alias in tenant A never affects tenant B [§3.7]

• Given tenant T1 whose alias map records {A, B} and tenant T2 that has the same template_ids A and B but no alias assertion
• When the querier compiles template_id.resolves_to(A) once for T1 and once for T2
• Then for T1 it expands to {A, B}
• And for T2 it expands to exactly {A}
• (Locks §3.7: alias sets are per-tenant; an assertion in one tenant is invisible to every other.)

Scenario RFC0001.15 — Retraction removes any member, including the canonical, and is itself audited

• Given tenant T whose alias map records the equivalence class {A, B} (A < B, so A is the derived canonical)
• When an operator retracts member A — the canonical / smallest member — from the class
• Then an alias_retracted audit event is durably recorded (same WAL-before-ack barrier and field shape as RFC0001.12) whose asserted set names A (here as representative_id, the operator’s anchor) plus an empty member_ids, and the actor
• And after the projection rebuilds, A is removed from the class, leaving {B} — a single member, which is no longer an alias set, so resolves_to(A) expands to exactly {A} and resolves_to(B) expands to exactly {B}
• (Locks the representative-independent retraction rule: retracting any member is well-defined even when it is the canonical/smallest; the canonical is re-derived as min of the remainder, and a class that drops to one member ceases to be an alias set.)

Scenario RFC0001.16 — A non-aliased id resolves to itself

• Given tenant T with leaf Z and no alias assertion naming Z
• When the querier compiles template_id.resolves_to(Z)
• Then the predicate expands to exactly {Z} — identical to the base-member behaviour and to bare template_id = Z (RFC0001.6)

6. Proposed design

The Ourios miner in detail. This is the section that the ourios-miner crate is implemented against; §5’s Acceptance criteria operationalise the commitments here, and §8 maps each §5 scenario to the technique that tests it.

Why §6 exists. Published Drain (§3) and Drain3 (§4) do not address the properties Ourios requires. Each row below is a gap this section closes:

6.1 Data model

Amendment 2026-05-13. Section rewritten to align the record schema with the OTLP LogRecord shape — the project’s stated ingest contract per docs/glossary.md (entry OTLP: “we do not invent our own format”). The investigation that surfaced the gap is docs/architecture/otlp-log-format.md. The pre-amendment schema treated logs as raw text strings; the amended schema treats every log as a structured OTLP record from the moment it enters the system. §6.2’s algorithm and its ingest signature were aligned to this amendment in a companion edit the same day (see §6.2’s amendment note below): the body.kind fork is at the top of the algorithm, the descent step incorporates the §6.1 template-key tuple, and the MinerCluster::ingest signature now takes a structured OtlpLogRecord rather than a raw &str.

The miner emits one record per ingested OTLP LogRecord. The record shape mirrors the wire shape of OTLP logs (the opentelemetry-proto LogRecord plus its inherited Resource and InstrumentationScope context) plus the miner-derived columns that template mining produces.

Record columns

The record carries three groups of columns. The OTLP-derived group preserves the structured shape the wire promised; the miner-derived group is what this RFC introduces; the reconstruction group exists only when the body was mineable (body.kind = String).

Identity and partitioning:

OTLP-derived columns (faithful to opentelemetry-proto):

Amendment 2026-06-11 — the effective timestamp lives in RFC 0005, not here. RFC 0005 §3.2 (amendment of the same date) adds a writer-derived effective_time_unix_nano Parquet column — time_unix_nano when non-zero, else observed_time_unix_nano, else 0 — following the OTLP logs data model’s recommendation (“Use Timestamp if it is present, otherwise use ObservedTimestamp”). The record shape above is unchanged: the miner emits no new field, the Parquet writer computes the column from the two timestamp fields already listed, and the wire time_unix_nano is stored verbatim including 0 — scenario RFC0001.10 (verbatim preservation) remains intact and normative. Time partitioning and the DSL time window key off the derived column (RFC 0005 §3.4 / RFC 0002 §6.2).

Body and miner-derived reconstruction:

Where:

• Param = { type_tag: ParamType, value: Bytes }. ParamType is one of IP, UUID, NUM, HEX, TS, PATH, STR, OVERFLOW. STR is the unmasked-wildcard fallback — used when a slot was created by template widening of a previously-fixed literal token (the literal itself becomes the param value); OVERFLOW carries (length: u32, sha256_prefix: [u8; 8]) instead of the original value (§6.5). params.len() == count(<*> in template), always (in the body_kind = String branch); §6.2 enforces this when a widening introduces new wildcard slots.
• Separator is a small inline byte string (typically 1–3 bytes in practice). Encoding in Parquet is an implementation detail that does not affect this RFC.
• KeyValue mirrors the OTLP KeyValue message: a key: String and a value: AnyValue. AnyValue is a discriminated union over string | bool | int | double | bytes | array | kvlist. Storing AnyValue faithfully in Parquet (rather than flattening to a string) is what keeps query expressions like attributes["client.address"] = "10.0.0.1" typed.
• BodyKind is a two-variant enum (String, Structured) — not the full AnyValue discriminator. The body column carries the encoded AnyValue payload; body_kind is the cheap routing flag the query planner uses to decide whether reconstruction is defined for this row.

Body representation (AnyValue handling)

OTLP’s LogRecord.body is AnyValue — string, bool, int, double, bytes, array, or kvlist. The spec is explicit (Logs Data Model §Body): “Body MUST support AnyValue to preserve the semantics of structured logs emitted by the applications.” Real OTel emitters send structured Body routinely, not just text.

Ourios distinguishes two body shapes at ingest:

• body_kind = String — LogRecord.body is AnyValue::String. The miner runs the §6.2 algorithm over the unwrapped string: tokenize, mask, descend the tree, attach to or create a leaf. params, separators, confidence, lossy_flag are populated per the existing semantics.
• body_kind = Structured — LogRecord.body is any other AnyValue variant (kvlist, array, int, double, bool, bytes). The miner does not run the §6.2 algorithm. The body is encoded with the Ourios canonical body encoding (see The Ourios canonical body encoding below) and stored in the body column; no template is mined, no params/separators are emitted. template_id is allocated per the Template-key composition rule below — for this branch the key is (severity_number, scope_name, BodyKind::Structured), so all structured-Body records sharing a (severity, scope) share one template_id. The leaf the id points at carries the Structured marker and an empty body_template. confidence = 1.0 (sentinel), lossy_flag = false (the canonically-encoded body is authoritative; nothing is reconstructed from a template).

This is the conservative default. It preserves the structural content of the body (the Ourios canonical body encoding below makes [§3.3] reconstruction well-defined for the structured branch: stored_bytes ↔ AnyValue is bidirectional and byte-deterministic), it avoids inventing template structure for arbitrary AnyValue trees, and it sidesteps the spec ambiguity of “what is the template for {"msg": "x", "user_id": 42}.” A future opt-in mine-inner-field mode (e.g., mine body.kvlist["msg"] as the line if present) is a configurable knob, not the default; that decision lives with the maturity-stage move from red → green once corpus evidence informs which inner-field conventions are worth specifying.

A third path — render-to-string + mine (canonicalise structured Body to JSON-ish text and run it through the §6.2 mining algorithm) — was rejected because mining over the JSON serialisation produces token templates that depend on the serialiser’s whitespace and field-ordering choices, which is both fragile (changing serialisers shifts every template) and defeats the §3.3 reconstruction guarantee for any record where the original wire form was protobuf rather than JSON. Storing the canonical encoding (without mining over it) is different from this rejected path: storage is faithful, it just doesn’t get a template extracted.

The Ourios canonical body encoding (body_kind = Structured).

Amendment 2026-06-09 (no canonical OTLP JSON exists). This paragraph previously called the encoding “the OTLP-canonical JSON encoding per the OTLP specification’s HTTP/JSON binding,” implying a spec-defined canonical form. There is none. Per the OTLP spec, the OTel common docs, and a maintainer answer (Josh Suereth, 2026-06-09): OTLP/JSON is the proto3 JSON mapping plus a short closed list of deviations (hex trace_id/span_id, integer enums, ignore-unknown-fields, lowerCamelCase) — with no normative rules on whitespace, key/field ordering, or number canonicalisation, and OTLP does not require lossless translation between formats. The text below is reframed to state the rule as an Ourios-local deterministic encoding, not an OTLP conformance point, and renamed to “the Ourios canonical body encoding.” No code and no RFC status change here; the encoder is ourios-core’s otlp::canonical (a separate follow-up PR aligns its doc comments).

Amendment 2026-06-11 (doubles round-trip bit-exactly — #130). The rule below previously left double precision implicit, and decode(encode(x)) drifted 1–2 ULP for ~12% of arbitrary finite f64. Investigating #130 located the loss on the decode side, not the encoder: the emitter already produces shortest-round-trip digits (serde_json’s Ryu f64 formatter — with-serde adds no custom double formatter), but serde_json’s default float parsing is approximate. The rule now pins both halves: doubles are emitted as shortest-round-trip JSON numbers and decoded with correctly-rounded float parsing (serde_json’s float_roundtrip feature, declared load-bearing in ourios-core), so the f64 round-trip is exact for every finite double — the faithfulness guarantee below holds for arbitrary doubles, not just “nice” ones. Non-finite doubles (NaN, ±∞) have no JSON-number form and encode as null — bytes that do not decode back; that pre-existing gap is explicitly out of scope here and stays open. No RFC status change.

The body column carries the Ourios canonical body encoding of the AnyValue: a proto3-JSON form, defined below. This is an Ourios-local deterministic convention, not an OTLP-mandated canonical form. OTLP defines no canonical or byte-deterministic JSON encoding. Its only normative JSON rules are the proto3 JSON mapping (per the protobuf spec) plus a short closed list of OTLP-specific deviations — trace_id / span_id as hex strings (not applicable to a body AnyValue, which carries no IDs), enum values as integers, ignore unknown fields, and lowerCamelCase field names. The spec is silent on whitespace, key/field ordering, and number canonicalisation, and OTLP does not require lossless translation between formats. There is therefore no “canonical OTLP JSON” to reference; the byte-stable encoding below is Ourios’s own, chosen so the body column is byte-deterministic (storage dedup) and the [§3.3] reconstruction guarantee is well-defined.

The concrete rule is the proto3 JSON mapping as emitted by opentelemetry-proto’s with-serde feature via serde_json:

• field names in lowerCamelCase;
• int64 / uint64 values as decimal strings (proto3 JSON’s canonical emit form; decoders accept a JSON number or string);
• double values as JSON numbers in shortest-round-trip form, decoded with correctly-rounded float parsing — decode(encode(x)) is bit-exact for every finite f64 (#130; see the 2026-06-11 amendment above);
• bytes as base64;
• KvlistValue and ArrayValue element order preserved as received — not sorted (this is explicitly not RFC 8785 / JCS canonical JSON);
• byte-deterministic because the proto types have a fixed serde field order and serde_json serialisation is deterministic.

Canonical byte examples: {"intValue":"-42"}, {"doubleValue":2.71}, {"boolValue":true}, {"bytesValue":"<base64>"}, {"arrayValue":{"values":[…]}}, {"kvlistValue":{"values":[{"key":"…","value":{…}}]}}.

“Deterministic” here means byte-identical. Re-encoding the same in-memory AnyValue with this encoder yields byte-for-byte identical output (within a fixed opentelemetry-proto / serde_json version). This is the byte-level reading, not a weaker struct-level one: it is what lets the body column be deduplicated and lets two receivers handling the same logical AnyValue produce the same stored bytes. The receiver path: OTLP/gRPC (protobuf wire) decodes to an in-memory AnyValue and re-encodes here; OTLP/HTTP+JSON decodes the incoming JSON to an in-memory AnyValue and re-encodes the same way, so the stored bytes do not depend on the producer’s whitespace, field order, or int64-as-number-vs-string choice.

The faithfulness guarantee — stored_bytes decode back to the original in-memory AnyValue — is an Ourios guarantee delivered by this encoder/decoder pair, not an OTLP lossless promise (OTLP makes none). lossy_flag = false for structured rows rests on this Ourios guarantee, not on any OTLP conformance claim.

Duplicate keys. OTLP KvlistValue is a repeated KeyValue that the data model treats as a map with unique keys; the data-model map equality is order-insensitive, but OTLP does not define wire-order equality, which is why preserving received order (above) is the safe, spec-permitted choice. A KvlistValue carrying duplicate keys is non-conforming OTLP input with no defined semantics. Ourios does not silently dedup or reorder such input: it preserves the entries verbatim in the encoding (so the round-trip stays faithful) and makes no map-semantic guarantee for it, rather than inventing one.

Template-key composition

A template’s identity (the discriminator the Drain tree uses to decide “is this the same template?”) depends on the body shape:

• body_kind = String — key tuple is (severity_number, scope_name, masked_body_tokens).
• body_kind = Structured — key tuple is (severity_number, scope_name, BodyKind::Structured). All structured-Body records sharing a (severity_number, scope_name) share one template_id. This intentionally forfeits structured-body shape clustering — the rationale is that the structured-Body branch’s value comes from the faithful preservation of attributes and the canonically-encoded body, not from grouping similar AnyValue shapes. Operators who need shape-level clustering can opt into a future body_shape_fingerprint column (a stable hash over the AnyValue’s structural skeleton — kvlist key-set, nested shape, leaf-type sequence; values ignored) as a reserved extension; the gate for adding it is “we have a concrete consumer,” not “it might be useful.”

The bullet rationale below applies to both branches:

• severity_number is part of the key because INFO and ERROR versions of the same body text are semantically distinct events. “user logged in” at INFO is a routine signal; “user logged in” at ERROR is an alarm (or an emitter bug) — collapsing them to one template_id would surface either as the other on query, which is a [§3.1] “no silent merges” violation in disguise. The OTLP-spec-valid severity_number = 0 (UNSPECIFIED) is a distinct key value, not coalesced with any specified severity.
• scope_name is part of the key because the same body text emitted from two different instrumentation scopes (myapp.login vs myapp.checkout) describes two different events. The scope is the OTel-canonical “which code path emitted this,” directly analogous to the package/logger name in traditional logging frameworks. Records with no scope (scope_name = None) cluster as their own (severity, None) bucket.
• resource_attributes are NOT part of the key. They identify who sent the record (service, host, k8s pod), not what event was emitted. The tenant_id derivation (below) already encodes the partition decision over Resource. Folding Resource into the template key would explode template cardinality proportionally to the deployment fleet size without adding semantic discrimination — the same myapp.login template from two replicas of service.name = api is the same template.
• event_name is not in the key today but is reserved as a candidate addition. RFC 0001 stays at the OTLP-canonical severity+scope key; promoting event_name into the key is a follow-up RFC patch once corpus evidence justifies it.

The Drain tree’s implementation of this tuple (extra prefix levels above length-N, tuple-keyed leaf lists, separate trees per (severity, scope), etc.) is §6.2 implementation territory and may be revisited based on cardinality observations from the corpus benchmark. The RFC pins only the semantic key.

Tenant derivation

tenant_id is derived per ResourceLogs group, not per OTLP export batch. Each ResourceLogs carries its own Resource.attributes, and a single OTLP export can contain multiple ResourceLogs groups from different sources — so one export can route records to multiple tenants. The derivation runs once per inherited Resource; the resulting tenant_id applies to every LogRecord under that ResourceLogs group (across all its ScopeLogs), and the receiver fans the records out into per-tenant streams.

The default per-Resource rule:

tenant_id := resource.attributes["service.name"]   if present
          ?: <operator-required fallback rule>

service.name is the conventional OTel unit of “what application emitted this,” and it maps directly onto Ourios’s per-tenant template-tree partitioning ([§3.7]). Operators with a different multi-tenant model (per-namespace, per-customer-id-attribute, composite of multiple attributes) configure an alternative rule; the receiver does not invent a tenant identity that the operator hasn’t declared.

If a ResourceLogs group’s Resource resolves to no tenant under either the default rule or the operator’s fallback, the receiver rejects the entire export batch with a controlled error (no panic, no silent assignment to a “default” tenant; the sender sees the failure and either fixes its emitter or its deployment). Per-Resource rejection within an otherwise-valid batch is not supported in this RFC — the all-or-nothing failure mode is simpler to reason about for the sender, and OTLP’s batch-level acknowledgement model fits all-or-nothing more naturally than partial-success. The receiver-side specification of this rejection path (and any future opt-in for partial acceptance) lives in RFC 0003 — OTLP receiver (forthcoming).

Template identity

template_id is a cluster-wide unique monotonic u64 (with each tenant seeing a monotonic subsequence), allocated when a new leaf is created and never reused or reassigned. The id space is shared across tenants so that the same u64 value never refers to two different leaves; the per-tenant subsequence guarantee preserves [§3.7] by making each tenant’s allocation order observable in isolation. Cross-tenant content identity is intentionally not guaranteed — two tenants emitting the structurally identical template (same (severity_number, scope_name, masked_body_tokens) tuple) will have different template_ids, so a template_id alone never links structurally-equivalent templates across tenants. (The u64 value itself is cluster-wide unique, per the previous paragraph; what is not guaranteed is that the same template across two tenants resolves to the same id.) This preserves [§3.7] (per-tenant template trees) by construction; cross-tenant analytics that need content identity (deduplication across tenants for storage savings, shared template dashboards) are an opt-in concern and are not provided by the miner. A future template_fingerprint side column may carry a canonical content hash over (severity_number, scope_name, masked_body_tokens) for opt-in cross-tenant use; the gate for adding it is “we have a concrete consumer,” not “it might be useful.”

Template version. template_version starts at 1 when the template is created and increments by 1 on every widening event: either a new wildcard slot opens (a previously fixed token at position i becomes <*>), or an existing wildcard’s typed parameter set changes (e.g. a <NUM> slot starts seeing <STR> values). To detect the second case, every leaf carries — alongside its template — a slot_types: Vec<HashSet<ParamType>> indexed by wildcard slot, recording every ParamType observed in that slot. A type expansion is the addition of a ParamType to one of these sets. The pair (template_id, template_version) uniquely identifies one structural state of a template. Queries against template_id = X return all versions; queries against (template_id, template_version) = (X, V) return only the named state. The DSL surface is RFC 0002’s concern, not this RFC’s, but the data model must support both.

Why two integers and not a content hash. A content hash makes identity global by construction; in a multi-tenant backend that is a tenant-isolation leak rather than a feature. A content hash also makes template_version redundant — once the canonical template string changes, the hash changes, so versioning collapses into alias-mapping between hashes. Per-tenant monotonic ints with an explicit version field are smaller in the Parquet column, easier to reason about under [§3.7], and keep (template_id, template_version) as a meaningful compound key.

6.2 Algorithm

Amendment 2026-05-13. Rewritten to take a structured OTLP LogRecord rather than a raw &str, in line with the §6.1 amendment. The algorithm now opens with the body.kind fork from §6.1’s Body representation: AnyValue::String runs the Drain mining steps (the prior algorithm, preserved verbatim below); every other AnyValue variant short-circuits to the structured emit per §6.1’s Template-key composition fork. Step 3’s descent now incorporates (severity_number, scope_name) into the tree key, again per §6.1 — the implementation choice (extra prefix layers, tuple-keyed leaf lists, separate trees per (severity, scope)) stays in §6.2 as the algorithm’s responsibility, but the semantic key is pinned by §6.1. The ingest signature on MinerCluster becomes ingest(record: &OtlpLogRecord); pre-amendment callers were ingest(tenant_id, raw: &str).

The miner sees an already-tenant-resolved (tenant_id, record: OtlpLogRecord) pair. The receiver (RFC 0003) is responsible for resolving tenant_id per ResourceLogs and fanning records into per-tenant streams before the miner sees them; §6.1’s Tenant derivation pins that contract.

For each ingested OTLP LogRecord:

0.  match record.body.kind:

      AnyValue::String(s):
          # Continue with the Drain mining algorithm in steps
          # 1–5 below, treating `s` as the `L_raw` of the prior
          # spec. body_kind = String.

      AnyValue::Bool | Int | Double | Bytes | Array | KVList:
          # Structured short-circuit per §6.1 *Body
          # representation*. The miner does NOT run the Drain
          # mining steps. body_kind = Structured.
          encoded = encode_canonical_body(record.body)
              # Ourios canonical body encoding (a proto3-JSON form;
              # lowerCamelCase fields, int64/uint64 as decimal
              # strings, bytes as base64, kvlist/array order
              # preserved — NOT sorted). This is an Ourios-local
              # deterministic convention, NOT an OTLP-mandated
              # canonical form: OTLP defines no canonical JSON and
              # requires no lossless translation. For records over
              # OTLP/gRPC the receiver decodes protobuf and
              # re-encodes here; for OTLP/HTTP+JSON it decodes to
              # the in-memory AnyValue and re-encodes the same way,
              # so stored bytes are byte-identical regardless of the
              # producer's whitespace / field order / int64 form.
              # The lossy_flag = false promise rests on this Ourios
              # round-trip guarantee — see §6.1 for the why.
          template_id = allocate_or_reuse_structured_template_id(
              record.severity_number,
              record.scope_name,
          )
              # Per §6.1 *Template-key composition*, the
              # structured-Body key is (severity_number,
              # scope_name, BodyKind::Structured). All structured
              # records sharing a (severity, scope) share one
              # template_id. The leaf the id points at carries the
              # `Structured` marker and an empty body_template.
          attach_structured(record, encoded, template_id,
                            confidence = 1.0,
                            lossy_flag = false)
              # confidence = 1.0 sentinel; lossy_flag = false
              # because the canonicalised body is authoritative,
              # nothing is reconstructed from a template.
          return

1.  L_tok, separators = tokenize(L_raw)
        # tokenize splits on Unicode whitespace only — every
        # codepoint matching `char::is_whitespace()` (ASCII space,
        # tab, CR, LF, plus the broader Unicode whitespace classes
        # U+0085, U+00A0, U+1680, U+2000–U+200A, U+2028, U+2029,
        # U+202F, U+205F, U+3000). Every other byte (including
        # punctuation such as `=`, `:`, `,`, `;`, `[`, `]`, `(`,
        # `)`) stays inside a token; structured separators are the
        # masking layer's responsibility (§4.2 / step 2). The
        # captured whitespace runs go into `separators` so that
        # reconstruction (§6.6) is byte-identical.
        # On failure (malformed UTF-8, embedded NUL, line longer
        # than max-line-bytes): emit a parse-failure record and
        # increment ourios.miner.parse_failures. Skip the rest.
        # Note: an empty-after-whitespace string (the AnyValue
        # carries `""` or only whitespace) is not a parse failure
        # — it has zero tokens and the miner short-circuits with
        # the cluster's `NO_TEMPLATE` sentinel rather than
        # descending the tree. The pre-amendment cluster code
        # already routes this case; the spec just records it.

2.  L_masked, typed_params = mask(L_tok)
        # mask applies the configured masking rules in order;
        # any token matching a rule is replaced with its type
        # tag (e.g. <IP>) and the original bytes are pushed
        # into typed_params with that tag. Unmasked tokens
        # remain literal.

3.  parent = tree.descend(record.severity_number,
                           record.scope_name,
                           len(L_masked),
                           L_masked[0..d-1])
        # Per §6.1 *Template-key composition*, the discriminator
        # for "is this the same template?" is the tuple
        # (severity_number, scope_name, masked_body_tokens).
        # Step 3 incorporates severity_number and scope_name into
        # the descent key alongside the masked-token prefix used
        # by published Drain. The implementation may layer extra
        # prefix levels above the length-N node, key leaf lists
        # by (severity, scope), or maintain separate trees per
        # (severity, scope) — the choice is cardinality-driven
        # and revisitable from corpus observations. The
        # severity_number = 0 (UNSPECIFIED) and scope_name = None
        # cases are valid distinct key positions; they cluster as
        # their own buckets, never coalesced with any specified
        # severity or named scope.
        # if a node along the path does not exist, create it.

4.  candidate = argmax over leaf in parent.leaves of
                  simSeq(L_masked, leaf.template)
    if candidate is None:
        # no leaves under parent yet; create one. Creation does not
        # emit an audit event — `ourios.miner.template.count` already reflects
        # leaf allocation, and §6.4 reserves the audit stream for
        # widening events whose semantics need cross-referencing.
        leaf = new Leaf(template = L_masked)
        parent.leaves.push(leaf)
        # On fresh-leaf creation the template is L_masked verbatim,
        # so every <*> in it came from mask(); params == typed_params.
        attach(L_masked, typed_params, separators, leaf,
               confidence = 1.0, lossy_flag = false)
        return

5.  similarity = simSeq(L_masked, candidate.template)
    confidence = similarity / threshold

    if similarity >= threshold:
        # clean or lossy attach; widen the template if needed.
        # widen() returns:
        #   widened           — the new template (existing fixed
        #                       positions that mismatched L_masked
        #                       become <*>)
        #   new_wildcards     — the set of positions that just
        #                       became <*> (the audit payload)
        widened, new_wildcards = widen(candidate.template, L_masked)
        if new_wildcards > 0:
            candidate.template = widened
            candidate.version += 1
            emit_audit(event_type = template_widened,
                       template_id = candidate.id,
                       old_version, new_version = candidate.version,
                       positions_widened = new_wildcards.positions,
                       ...)
            ourios.miner.merges.inc()

        # Build the params array. One entry per <*> in the (possibly
        # just-widened) template, in template order. For each slot:
        #   - if the slot existed before this attach AND mask() emitted
        #     a typed_params entry for it, use that entry verbatim.
        #   - if the slot is a fresh wildcard from this widening (the
        #     position held a literal token in candidate.template before
        #     the widen call), the original literal at that position in
        #     L_tok is captured as { type_tag: STR, value: L_tok[pos] }.
        # Without this step §6.1's "one entry per <*> slot" invariant
        # is violated and §6.6's reconstruct() has no value to insert
        # at the freshly-widened position.
        params = build_params(candidate.template, typed_params,
                              L_tok, new_wildcards)

        # Type-expansion: if any wildcard slot now sees a typed param
        # whose type tag is not already in that slot's observed-type
        # set, widen the slot's type set, bump the version, and
        # audit. The leaf carries `slot_types: Vec<HashSet<ParamType>>`
        # alongside its template (data model in §6.1).
        new_types = update_slot_types(candidate, typed_params)
        if not new_types.is_empty():
            candidate.version += 1
            emit_audit(event_type = template_type_expanded,
                       template_id = candidate.id,
                       old_version, new_version = candidate.version,
                       slots_expanded = new_types,
                       ...)

        attach(L_masked, params, separators, candidate,
               confidence,
               lossy_flag = false)  # §6.6: lossy_flag is set only on
                                    # tokenizer/preprocessing failure,
                                    # not on confidence < 1.0
        return

    if similarity >= floor:
        # lossy zone: the line is "close" but doesn't meet
        # threshold. Create a new leaf rather than force-merging.
        # Body retention is unconditional in this branch.
        leaf = new Leaf(template = L_masked)
        parent.leaves.push(leaf)
        # As in the candidate-is-None branch, the new leaf's template
        # is L_masked verbatim, so params == typed_params.
        attach(L_masked, typed_params, separators, leaf,
               confidence,
               lossy_flag = false,
               body = L_raw)  # forced body retention
        ourios.miner.body_retention.utilization.observe(retained = true)
        return

    # similarity < floor: parse failure
    ourios.miner.parse_failures.inc()
    emit_failure_record(L_raw, reason = "no candidate above floor")

Branching invariants:

• Step 0’s structured short-circuit never enters the Drain mining steps (1–5). Structured-Body records do not widen, do not emit template_widened or template_type_expanded audit events, do not contribute to ourios.miner.merges, and never carry params/separators. The structured branch’s [§3.1] preservation is vacuous: no template merge happens, so no silent merge is possible.
• The tree only deepens on first observation of a (severity_number, scope_name, length, prefix tokens) shape (the §6.1 template-key tuple, anchored at this section’s step 3).
• Leaves are split (new leaf created) when the best candidate is in the lossy zone; they are never split when the candidate is clean.
• A leaf is widened (wildcards introduced) when a clean attach would otherwise mismatch positions. Every widening emits an audit event (§6.4).
• A leaf’s wildcard slot is type-expanded when an attach maps a typed parameter whose ParamType is not already in that slot’s slot_types[slot] set. Type expansion increments template_version and emits a template_type_expanded audit event (§6.4).
• A single attach can trigger both wildcard-widening and type-expansion in the same leaf; in that case template_version increments twice and two audit events are emitted, in that order.
• The leaf’s template_version only increments on widening or type-expansion, not on a clean attach. Structured-Body leaves are never widened or type-expanded; their template_version stays at 1 for the lifetime of the leaf.

6.3 Confidence scoring [§3.1]

confidence = simSeq / threshold. The ratio framing makes the decision boundary land at confidence == 1.0 regardless of the configured threshold, which gives ourios.miner.confidence.p50 and ourios.miner.confidence.p01 ([§3.1] required metrics) a stable interpretation across tenants with different thresholds: the p01 value tells you how close the bottom 1% of attaches are to the merge boundary. A collapsing p01 means many lines are barely passing — a tuning signal even though the threshold itself has not moved.

Three zones, with concrete defaults:

• confidence ≥ 1.0 (i.e. simSeq ≥ threshold): clean attach. No body retention.
• floor / threshold ≤ confidence < 1.0: lossy zone. The line attaches to a freshly created leaf rather than being force-merged into the candidate (see §6.2 step 5). body is retained unconditionally; lossy_flag follows the §6.6 rule (set only on reconstruction failure, not on lossy zone alone — the body is available either way).
• confidence < floor / threshold: parse failure. ourios.miner.parse_failures increments; the line is written to a failure record with the original bytes intact.

Defaults. threshold = 0.7, floor = 0.4. The threshold floor is fixed by [§3.1] (“threshold ≥ 0.7, lowering requires an RFC, not a config change”); the lossy-zone floor is a tuning knob between threshold and 0. floor = 0.4 matches the paper’s reported default threshold, on the reasoning that lines below the paper’s own bar are likely genuinely different events. Tuning the floor is a per-tenant config decision; it is not load-bearing for any invariant.

6.4 Merge policy [§3.1]

A template widening (per §6.2 step 5) is the operation that [§3.1] calls a “merge.” Every widening emits an audit event with the schema:

{
  event_type: AuditEventType,  # enum:
                               #   template_widened
                               #   template_type_expanded
                               #   template_widening_rejected_degenerate
  tenant_id: TenantId,
  template_id: u64,
  old_version: u32,
  new_version: u32,
  old_template: String,        # canonical form, with <*> for wildcards
  new_template: String,
  triggering_line_hash: [u8; 16],  # blake3 of L_raw, for cross-ref
  triggering_line_sample: Option<String>,  # first 256 B of L_raw
  positions_widened: Vec<u16>, # token positions that became <*>
                               # (empty for template_type_expanded)
  slots_expanded: Vec<SlotExpansion>,
                               # slot index + newly added ParamType(s)
                               # (empty for template_widened)
  timestamp: SystemTime,
}

event_type is the field §6.7’s drift-detection query filters on. ourios.miner.merges increments on every event whose event_type is template_widened or template_type_expanded (the two structural widenings); template_widening_rejected_degenerate is recorded but does not increment ourios.miner.merges. The audit stream is written to the same WAL as the data records and ends up in a dedicated audit Parquet file per tenant per compaction window (schema in ourios-parquet’s RFC, not this one).

Default policy: strict. Widening is permitted whenever the clean-attach path in §6.2 would otherwise mismatch positions. The audit event is mandatory — no widening, of any reason, ever proceeds without one. Code paths that would emit a widening without emitting an audit event are blocked at PR review per hazards.md H1.

WAL durability ordering of audit events. A single attach may emit two audit events in order (RFC0001.7: template_widened immediately followed by template_type_expanded) and one data record. The contract this RFC requires from the future ourios-wal RFC is an ordering-plus-durability-barrier: a data record carrying template_version = V must not become durable before every audit event justifying the leaf’s progression to V is durable. Crash recovery may then observe some prefix of [event_1, event_2, …, data_record], but never a state in which the data record exists without the events that caused its version stamp. Any framing strategy that satisfies this — a composite multi-record frame, batched-fsync ordering, a two-phase write-then-link, anything else — is acceptable; the framing is ourios-wal’s choice, the ordering barrier is this RFC’s requirement. Without it, replay would bump template_version fewer times than the in-memory leaf did and the surviving data records would reference a version the audit stream cannot substantiate.

Degenerate template guard. If a widening would leave the template with zero non-wildcard tokens (the entire template becomes <*> <*> … <*>), the widening is rejected, the line is treated as a parse failure (confidence = 0, retain body, increment ourios.miner.parse_failures), and an audit event with event_type = template_widening_rejected_degenerate records the rejection. A fully-wildcard template provides no parsing value and would swallow arbitrary lines.

6.5 Parameter handling [§3.2]

Per-parameter byte limit. Default 256 B, configurable up to 1 KiB (the [§3.2] ceiling). Above 1 KiB requires an RFC.

Overflow behaviour. When a parameter value (post-masking) exceeds the configured limit, the parameter slot is replaced by an OVERFLOW marker:

Param {
  type_tag: ParamType::OVERFLOW,
  value: encode(length: u32, sha256_prefix: [u8; 8]),
}

The original line L_raw is captured into the body column unconditionally (overflow forces body retention, regardless of lossy_flag). The 8-byte SHA-256 prefix lets queries “find rows where this exact long parameter occurred” without storing the long value in the columnar data. Reconstruction honours overflow: reconstruct(record) falls back to body when any param has type_tag == OVERFLOW.

Telemetry. Two metrics for [§3.2] and hazard H2:

• ourios.miner.params.overflow (counter, attributes ourios.tenant, ourios.service): increments per overflow.
• ourios.miner.params.overflow.utilization (gauge, attributes ourios.tenant, ourios.service): rolling overflow rate. Alert at > 0.01 per service per [§3.2].

6.6 Body reconstruction [§3.3]

Amendment 2026-06-08 (reader render contract). H7.3 (§5) referenced “the §6.6 warning marker,” but §6.6 defined no such marker — the prose Reader behaviour paragraph it replaced described one only informally. This amendment adds the Reader render contract subsection below, defining the marker as a structured, out-of-band per-row Reconstruction signal (Faithful | RetainedVerbatim) the reader attaches to the rendered row — never a mutation of the body bytes — and pinning the lossy short-circuit H7.3 requires (return body verbatim, do not call reconstruct). The clean-path read-time template lookup (a registry mapping (template_id, template_version) → tokens at read time) is explicitly out of scope and deferred to the querier’s reader-materialisation story (RFC 0007). RFC 0001 stays specified; this clarifies the §6.6 contract, it does not change the on-disk schema or the mining algorithm.

Capture, always. Every successful tokenization in §6.2 step 1 populates the separators array with the bytes between adjacent tokens (and the leading and trailing bytes of the line). The array length is tokens.len() + 1. There is no whitespace heuristic and no “is this whitespace trivial” decision — the bytes are captured verbatim. Storage cost is bounded (typical separator is one space; the array dictionary-encodes well in Parquet) and the implementation has no fuzzy boundary that could decide to drop bytes silently.

Reconstruction function.

fn reconstruct(record: &Record) -> Bytes {
    if record.lossy_flag {
        return record.body.expect("lossy implies retained body");
    }
    if record.params.iter().any(|p| p.type_tag == OVERFLOW) {
        return record.body.expect("overflow implies retained body");
    }
    let template = lookup(record.template_id, record.template_version);
    let mut out = BytesMut::new();
    out.extend_from_slice(&record.separators[0]);
    for (i, token) in template.tokens.iter().enumerate() {
        match token {
            Token::Fixed(s) => out.extend_from_slice(s),
            Token::Wildcard(slot) => {
                out.extend_from_slice(&record.params[slot].value)
            }
        }
        out.extend_from_slice(&record.separators[i + 1]);
    }
    out.freeze()
}

lossy_flag semantics. Set to true if and only if reconstruction is not guaranteed to equal the ingested bytes:

• The tokenizer failed (malformed UTF-8 inside a token, embedded NUL, line exceeded the configured max_line_bytes cap before tokenization completed).
• A preprocessing rule explicitly rejected the line.

The lossy zone in §6.3 (low confidence) does not automatically set lossy_flag: the body is retained either way, and reconstruction from template + params + separators is still expected to match. The flag is reserved for the cases where the record genuinely cannot be reconstructed.

Reader render contract

The functions above run at write time: reconstruct is the property-test oracle and the in-process renderer the miner exercises while the template is in hand. The reader — the read-side path that materialises a stored Parquet row back into the effective original line for a query result — is a distinct caller, and H7.3 pins the contract it must honour. The contract below covers String-body rows (body_kind = String); structured bodies are out of scope (see the end of this subsection).

For a String-body row, the render result is (bytes, reconstruction). Rendering yields two things: the effective original line bytes and a per-row reconstruction signal

#![allow(unused)]
fn main() {
enum Reconstruction {
    Faithful,         // bytes == ingested line, reconstructed from template
    RetainedVerbatim, // bytes are the retained `body` column, not reconstructed
}
}

The Reconstruction signal is the “§6.6 warning marker” that H7.3 references. It is structured, out-of-band metadata attached to the rendered row — not a mutation of the body bytes. A consumer (the DSL output layer, RFC 0007; a UI) renders RetainedVerbatim as a “rendered from the retained body bytes, not reconstructed” warning beside the row, exactly as it would render any other per-row annotation.

A body-byte annotation is explicitly rejected as the marker. Prefixing the body with a sentinel string, wrapping it in a marker character, or otherwise editing the bytes to carry the warning would break the verbatim guarantee: the whole point of the lossy path is that an operator asking “show me what was actually logged” gets the ingested bytes back unchanged [§3.3]. The marker therefore lives beside the bytes, never inside them.

Lossy path (H7.3). For a row with lossy_flag = true — the tokenizer-failure / explicit-rejection cases enumerated under lossy_flag semantics above — the reader returns the body column verbatim with Reconstruction::RetainedVerbatim, and does not invoke reconstruct: no template lookup, no token walk. The same short-circuit applies to a String-body row carrying an OVERFLOW param (§6.5): its retained body is returned verbatim with Reconstruction::RetainedVerbatim. reconstruct’s own lossy_flag / OVERFLOW early returns remain in place as a defensive guard for callers that reach it anyway; the reader’s contract is to short-circuit before that call, so the guard is belt-and-braces, not the primary mechanism.

Structured bodies are out of scope of this amendment. A body_kind = Structured row renders its body column from the Ourios canonical body encoding (the §6.1 rule, exercised by the RFC0001.9 structured short-circuit). This amendment defines the Reconstruction marker for the implemented String path only; how a structured render maps to a Reconstruction signal is left open here, to be settled when structured-body rendering is wired.

Clean path. For a faithful row (lossy_flag = false, no OVERFLOW param, template available) the reader invokes reconstruct(record) and attaches Reconstruction::Faithful. This path requires resolving (template_id, template_version) → tokens against a template registry available at read time. That registry — how the reader obtains the template a stored row was mined against, given that the miner’s in-memory tree is an ingest-side structure — is a separate concern and out of scope of this amendment. Today reconstruct is exercised only where the template is already in hand (the write-side property test H7.1, and H7.4); the read-time lookup mechanism is future work, tracked with the querier’s reader-materialisation story (RFC 0007). This amendment pins the render contract and the lossy path; it does not specify a template-registry design.

The reader never silently substitutes one rendering for the other: every rendered String-body row carries its Reconstruction signal, and the lossy/clean branch is selected solely by the row’s lossy_flag (and the §6.5 OVERFLOW case), never inferred from the bytes.

Property test. For every row r in the corpus where r.lossy_flag == false:

reconstruct(r) == ingested_bytes(r)

Failure is a build break, not a regression — [§3.3] and hazard H7 both name this as the property test that gates merges.

6.7 Template versioning and drift [§3.5], hazard H5

A template’s structural state changes over time as widenings (§6.4) and parameter-type expansions accrue. Each change increments template_version and emits an audit event of type template_widened or template_type_expanded. ourios.miner.template.version_changes counts these.

Two distinct cross-cutting questions. “Same leaf, different structural snapshots” and “different leaves that mean the same thing” are separate problems, with separate query forms:

• Cross-version (within one leaf). A leaf’s template_id is stable across every widening of that leaf (§6.1, “Template identity”); only template_version advances. So a literal predicate where template_id = X already returns rows from every version of leaf X by construction — no alias resolution required. To pin to a single structural snapshot, query where (template_id, template_version) = (X, V).
• Cross-alias (across leaves). When a deploy changes a log line enough that the miner allocates a new leaf instead of widening the existing one, the operator has two template_ids for what is semantically the same template. Resolving “all rows for the thing X represents” then requires walking an alias set that spans leaves. RFC 0002 §5.4 exposes this as where template_id.resolves_to(X); bare where template_id = X does not follow alias chains.

The data model in §6.1 supports both shapes: cross-version is free because template_id is stable across widenings; cross-alias is served by a separate alias index that maps a representative template_id to the equivalence class of template_ids the operator (or a future inference layer) considers semantically the same template.

Alias index lifecycle. Cross-alias is structurally distinct from cross-version: widening is intra-leaf and increments template_version (one template_id, several versions); aliasing is inter-leaf and groups template_ids the miner allocated separately (different template_ids, the operator asserts they mean the same thing). template_widened events therefore do not populate the alias index — they live on the cross-version axis.

Alias write path: operator-driven and audited [§3.1]. The alias index is operator-driven, never silently inferred. An alias assertion — “leaf B is the same template as leaf A” — is an explicit operator action recorded as an audited, durable event, exactly as §3.1 requires of any merge (“every merge emits an audit event; explicit”). Automatic inference is the precise failure mode §3.1 forbids: an auto-aliased cross-semantic merge is a silent merge by another name. The amendment below replaces the previous “no creation event / produced out of band” deferral with the mechanism.

Amendment (alias write path, 2026-06-07). Resolves the §9 open question “Alias index creation mechanism.” The alias index is produced by operator-driven, audited, reversible assertions on the §6.4 audit stream; automatic inference is deferred to a possible future propose → operator-confirm layer (proposals never enter the active index unconfirmed). Adds the alias_asserted / alias_retracted audit events, the per-tenant alias-map projection the querier reads, and §5 scenarios RFC0001.12–RFC0001.16. The RFC re-enters the ladder at specified until those scenarios land (precedent: the RFC 0003 served-binary amendment).

The alias model. An alias set is an equivalence class of template_ids within one tenant [§3.7]. Membership — which template_ids are in the class — is the only thing that carries contract weight: resolves_to expands by membership and nothing else (RFC0001.13). The class also has a canonical representative, defined as the numerically smallest member of the materialized set. The canonical representative is a derived display/identity convenience, not what defines membership: it gives the set a stable name and is deterministic regardless of assertion order, but it plays no part in deciding who belongs to the class. This derivation rule is an evolvable implementation detail, not a contract — it may change (for example, letting an operator designate a preferred representative) without changing alias-set semantics, precisely because resolves_to expands by membership regardless of which member is canonical. (This is distinct from the event-level representative_id below, which is merely the operator’s anchor id for an assertion and need not equal the derived canonical.) Membership is cross-leaf only: it groups template_ids the miner allocated as separate leaves. It never crosses the cross-version axis — every version of a single leaf already shares one template_id (§6.1, “Template identity”), so template_version is not an alias concern and the alias index holds no template_version field. The two axes stay disjoint: widenings move a leaf along template_version; aliasing groups distinct template_ids.

The assertion event. Aliasing is expressed by two new AuditEventType variants on the §6.4 audit stream, carrying an alias-specific payload:

{
  event_type: AuditEventType,  # alias_asserted | alias_retracted
  tenant_id: TenantId,
  representative_id: u64,       # operator's anchor id for this
                               # assertion — names one member of the
                               # asserted set; carries no contract
                               # weight beyond that and need not equal
                               # the set's derived canonical (smallest)
  member_ids: Vec<u64>,        # the other ids grouped by / removed
                               # from the set in this assertion
  actor: ActorId,              # operator / API principal that
                               # issued the assertion — aliasing is
                               # never anonymous (§3.1 "explicit")
  reason: Option<String>,      # operator-supplied justification
                               # (e.g. "deploy 2026-06 re-split the
                               # login template"), <= 256 B
  timestamp: SystemTime,
}

The asserted set of an event is the full union {representative_id} ∪ member_ids — representative_id is one named member of that set, never excluded from it. An alias_asserted event groups its entire asserted set into one equivalence class; an alias_retracted event removes every id in its asserted set from their class. Membership is therefore defined purely by the union of ids in the event, independent of which id the operator chose as the anchor.

These events flow through the same audit stream as template_widened (§6.4) and inherit its durability contract: they are written to the WAL and become durable under the §3.4 WAL-before-ack barrier before the assertion is acknowledged to the operator. An assertion that has not hit the WAL is not acknowledged — there are no in-memory-only aliases. Unlike widenings, alias events are not emitted on the ingest hot path; they originate from an explicit operator/control-plane call, so they do not gate attach latency. They do not increment ourios.miner.merges (that counter is reserved for the two structural widenings, §6.4); alias activity is counted separately as the ourios.miner.alias.assertions / ourios.miner.alias.retractions counters enumerated in §6.8’s telemetry table (mandatory, ourios.tenant attribute), keeping operator-driven aliasing first-class telemetry per [§3.1] / §6.3 alongside ourios.miner.merges.

Materialization and storage. The durable alias event log (the alias_asserted / alias_retracted stream above) is the source of truth; the queryable per-tenant alias map is a projection built by folding that log per tenant. Each alias_asserted unions its full asserted set ({representative_id} ∪ member_ids) into one equivalence class, merging any pre-existing classes that share a member so that overlapping assertions converge on a single class regardless of arrival order; each alias_retracted removes its asserted set’s ids from their class. The canonical representative of each materialized class is then derived as min(members) — so it re-derives automatically when membership changes and never depends on which id any event named as its anchor. A class that drops to a single member is no longer an alias set (that lone id resolves only to itself, RFC0001.16). The folded result is persisted as a per-tenant artifact (one map per tenant, not per partition) that the querier reads at compile time — note: that persisted artifact is the deferred cache of the 2026-06-12 amendment below; in v1 the querier folds the map directly from the audit stream and no artifact exists yet. Three venues were considered:

• Per-tenant projection from the audit/alias event log (chosen). Aliases are a tenant-scoped projection rebuilt from the durable event log and persisted as a small per-tenant alias-map artifact under the tenant root. This reuses the §6.4 audit infrastructure end-to-end (no new write plane), matches the tenant scope of the data (§3.7), and keeps the truth (the event log) append-only and replayable while the map is a cache that can always be rebuilt by re-folding the log.
• Tenant-root aliases.parquet / JSON written directly (rejected as the source of truth). A directly-mutated file has no audit trail of its own; it would either duplicate the event log or become an unaudited mutation point, violating §3.1’s “every merge is audited.” It survives only as the serialization format of the projection above — a storage-layer detail (RFC 0005), not the model.
• Extending the partition Manifest (rejected). The manifest is partition-scoped (Manifest { generation, files }, one per partition); alias sets are tenant-global. Putting a tenant-global structure in a partition-local manifest would fragment one logical map across every partition and force N-partition fan-in on every query. Poor fit.

Eventual consistency / read staleness. The querier and the ingester/control-plane run in separate processes (§6.6 names the same seam for the live template registry). The alias map the querier reads is therefore eventually consistent with the latest assertion: an alias asserted or retracted at t becomes visible to queries once the projection is rebuilt and republished, not instantaneously. This is the same staleness window the §6.6 querier↔registry seam already accepts. Staleness is bounded in both directions by the projection refresh / snapshot cadence:

• A not-yet-visible assertion makes resolves_to(X) return a subset of the eventual membership — it never returns rows from a set the operator did not yet assert, so it can only temporarily under-include.
• A not-yet-visible retraction leaves a stale projection still expanding to the old, larger set, so resolves_to(X) can temporarily over-include a member the operator already removed.

Both are transient and self-correcting on the next projection rebuild, and neither cross-contaminates across tenants (§3.7) or fabricates a grouping no operator ever asserted — the over-inclusion is always a previously asserted membership, never a phantom one. The bound on the window (snapshot cadence) is a storage/serving-layer knob, deferred to the RFC 0005 storage decision (see §9).

Amendment 2026-06-12 — the storage decision is made (v1), by the RFC 0005 line. alias_asserted / alias_retracted persist as event_kind 4 / 5 on the RFC 0005 §3.7 audit stream, and in v1 the querier derives the per-tenant alias map at query-compile time by scanning the tenant’s audit stream for those kinds and folding them through this section’s projection semantics (RFC 0005 §3.7.1) — there is no persisted map artifact yet, so “the projection is rebuilt and republished” above reads, in v1, “the audit events are durably written and flushed”; the staleness bound is audit-flush visibility rather than a snapshot/rebuild cadence, with the same bounded under-/over-inclusion directions. The cached per-tenant artifact (file/format/cadence) stays deferred behind the RFC 0009 §3.4 manifest fork; because the event log remains the source of truth, adding the cache later changes no query-visible semantics — the same v1-full-replay-now / accelerate-later shape as §6.9’s snapshot.

Reader / query contract. RFC 0002’s template_id.resolves_to(X) (RFC0002.9, §5.4) loads the requesting tenant’s alias map at compile time and expands X to its alias-set membership: template_id IN {X} ∪ members(set containing X). With no assertions for the tenant, or for an X in no set, the membership is exactly {X} — identical to today’s base-member stub and to bare template_id = X for that id (RFC0001.6 still holds: bare equality never follows alias chains). Expansion is by the set, not by direction: passing any member of a set (representative or not) resolves to the whole set.

Reversibility. An operator can retract an alias; the retraction is itself an alias_retracted audit event with the same durability and isolation guarantees, and the projection drops the retracted membership on its next rebuild. Aliasing and un-aliasing are both explicit, both audited, never silent — closing the loop with §3.1.

sequenceDiagram
    participant Op as Operator / control plane
    participant WAL as WAL (§3.4 barrier)
    participant Log as Alias event log (§6.4 stream)
    participant Proj as Per-tenant alias-map projection
    participant Q as Querier (resolves_to)
    Op->>WAL: assert alias {rep, members, actor, reason}
    WAL-->>Op: ack (only after fsync — §3.4)
    WAL->>Log: append alias_asserted (durable)
    Log->>Proj: fold per tenant (rebuild + publish)
    Q->>Proj: load tenant alias map at compile
    Q->>Q: resolves_to(X) -> template_id IN members(set with X)
    Note over Op,Proj: retraction is the symmetric alias_retracted event

Future work — automatic inference. A later layer may propose aliases from a post-deploy heuristic (e.g. a burst of template_widened / fresh-leaf allocations correlated with a deploy timestamp). Such proposals are never written to the active alias index directly; they enter a review queue and become real only via the same operator-confirmed alias_asserted event specified above. This RFC does not specify that layer — see §9.

Drift detection as a first-class query. “Templates that gained a new version in the window [t1, t2]” is a query against the audit event stream:

SELECT template_id, MIN(old_version), MAX(new_version),
       COUNT(*) AS widening_count,
       MIN(timestamp), MAX(timestamp)
FROM template_audit
WHERE event_type IN ('template_widened', 'template_type_expanded')
  AND timestamp BETWEEN $t1 AND $t2
GROUP BY template_id
ORDER BY widening_count DESC

(SQL shown for spec clarity; the user-visible form is the RFC 0002 DSL, not raw SQL — see hazard H6.) Operators use this query after deploys to spot templates whose structure changed; a sudden cluster of template_widened events correlated with a deploy timestamp is exactly the H5 detection signal.

6.8 Telemetry [§3.1], §6.3

Amendment 2026-06-03. Telemetry export is realigned from a Prometheus client/scrape model to the OpenTelemetry SDK (the maintainer direction recorded against RFC 0009 §3.6 and the roadmap §5 note). This amendment fixes the export architecture and the Prometheus-era terminology (registry → meter provider, scrape → OTLP push, labels → attributes) throughout §§6.8–6.9 and the §5 scenarios.

Amendment 2026-06-08 (dotted-semconv migration — landed). The metric and attribute names are now the dotted-ourios.miner.* scheme, defined in the semconv/registry/ weaver registry alongside the compaction set (RFC 0009 §3.6) and consumed through the generated ourios-semconv constants. The table below lists the registry names. Instrument kinds are unchanged from the original set; the confidence.p50 / confidence.p01 gauges remain in-process views of the confidence histogram (RFC0001.8) — the question of whether they become collector-/backend-derived quantiles over the exported histogram is a genuine contract change to the §3.1.2 mandatory set and stays deferred to its own review, not folded into this rename.

The registry names were audited against the OpenTelemetry semantic-conventions naming rules (the canonical docs/general/{naming,metrics}.md plus the weaver registry policies): counters drop the Prometheus _total suffix and take singular {annotation} units; the two fraction-of-total gauges use the conventional .utilization segment (unit 1) rather than a bespoke .ratio; and the per-line elapsed-time histogram is ourios.miner.duration (UCUM s), the conventional segment for a discrete operation’s elapsed time, not latency.

Export architecture (OTel SDK + OTLP)

Metrics are instrumented through the OpenTelemetry meter API and exported via the OTel SDK’s OTLP metric exporter (push, over OTLP to a collector / endpoint). There is no prometheus client crate and no /metrics scrape endpoint; any Prometheus compatibility is a downstream collector concern, not Ourios’s.

The dependency split follows the standard OTel layering so the heavy SDK and transport crates do not leak into every library:

• Instrumented crates (ourios-miner, ourios-parquet, ourios-ingester, ourios-querier) depend only on the lightweight opentelemetry API crate and resolve instruments through global::meter("ourios.<subsystem>"). No SDK, no OTLP, no transport dependency in a library crate.
• A new ourios-telemetry crate owns the heavy deps — the opentelemetry_sdk and opentelemetry-otlp crates (the upstream package names, underscore and hyphen respectively) plus the OTLP transport. It exposes an init() that builds the OTLP push MeterProvider (periodic-reader export, interval configurable), installs it as the process-global provider, and returns a guard whose shutdown() flushes pending metrics on exit. The binary (ourios-server) calls init() once at start-up; benches and integration tests call the same entry point or substitute an in-memory reader. Adding this crate extends the CLAUDE.md §7 target layout; the new-crate commitment is blessed here, in this RFC, per §7’s rule.

Dimensions are OTel attributes, not Prometheus labels, and OTel splits them in two: resource attributes identify the telemetry producer and are set once on the MeterProvider; data-point attributes vary per measurement. Ourios’s own identity — service.name = ourios-<role> (e.g. ourios-ingester, ourios-querier, matching the role the ourios-telemetry crate initialises the provider for; with service.version, etc.) — is a resource attribute: per the semantic conventions it MUST be set once on the provider’s Resource and MUST NOT be repeated on individual data points.

The per-measurement dimensions in the table below — among them ourios.tenant, the originating service of the ingested logs, and per-metric dimensions like event_type — are data-point attributes. A single ingester multiplexes many tenants and many source services, and [§3.1] / [§3.2] require per-(tenant, service) breakdowns — notably the §6.5 / H2.2 per-service overflow alert — which a single producer-level resource attribute could not provide. The service dimension here is the log’s source service (the value §6.1’s tenant derivation reads), distinct from Ourios’s own service.name — it must not reuse that reserved resource key. It is exported under the dedicated ourios.service attribute key; the tenant dimension is ourios.tenant and the merges change-kind is ourios.miner.template_change (registry enum widened / type_expanded). All three are defined in semconv/registry/attributes.yaml and consumed through the generated ourios-semconv constants.

The mandatory set is defined by the semconv registry: the ourios.miner.* entries in semconv/registry/, surfaced as the generated ourios_semconv::OURIOS_MINER_* constants, are the source of truth for which metrics the miner must expose. Each is registered as an instrument on the ourios.miner meter when the miner is constructed.

OTel’s metric model is collect-on-read: a reader / exporter sees the data points produced during a collection cycle, and an instrument contributes a data point on its first real measurement. The miner emits no synthetic zero-traffic points — so every exported series carries the registry’s required attributes, with no sentinel attribute value to collide with a real tenant / service or violate the template_change enum. §3.1.2 is verified by exercising every instrument with a small representative workload and collecting the metric stream (an SDK in-memory reader in tests): the registry pins the mandatory set; the collection proves each instrument is registered and emits real data under the required attributes.

The metrics enumerated in [§3.1] are mandatory. Full set (the dotted-ourios.miner.* registry names; the dimensions shown are exported as attributes, not labels — tenant is ourios.tenant, service is ourios.service):

ourios.miner.confidence.p50 and ourios.miner.confidence.p01. The ourios.miner.confidence histogram is the source of truth; the two gauges are convenient named views derived from it in-process. The miner recomputes them on a short ticker (default 10 s, configurable; the cost is one quantile evaluation over the histogram per tenant per service per tick — negligible relative to the hot path) and caches the value between ticks so a metric export cycle never blocks on recomputation. The gauges exist so alerting rules and runbooks can name them directly per [§3.1] rather than spelling out a histogram_quantile(...) expression at every reference.

The histogram bucket boundaries are tuned to straddle the decision boundary at 1.0 (see §6.3): default buckets [0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 1.0, 1.05, 1.2, 1.5, 2.0, +Inf].

6.9 Persistence and recovery

Amendment 2026-06-10 (snapshot store / cadence / scope resolved). The three §9 open questions deferred from this section — target store, cadence, and scope — are now pinned. The snapshot is a rebuildable recovery-acceleration cache, not durable state (the WAL is the truth per [§3.4]), which is what licenses the choices: local disk, WAL-adjacent (object storage deferred, see §9); per WAL-segment-rotation cadence, recording the WAL high-water mark; per-tenant scope. The format is a leading u8 version byte then a payload; recovery dispatches on byte 0. In v1, recovery replays the full WAL in both the known- and unknown-version branches, because the resume-from- high-water-mark optimisation needs the RFC 0008 §6.7 checkpoint / replay-from-offset API, which is not yet implemented; the format records the high-water mark so the optimisation can be switched on later without a format change. This makes the §3.5.1 / §3.5.2 acceptance criteria buildable. The matching §9 entries are marked RESOLVED.

Amendment 2026-06-12 (v2 — restore switched on). The RFC 0008 §6.7 checkpoint / offset-carrying-sink API this section was gated on is now specified to land (RFC 0008’s same-day §6.1/§6.7 amendment is the other half of this design), so the known-version branch of the recovery algorithm below restores the tree and replays only the WAL tail above the snapshot’s recorded high-water mark S — exactly as written in step (2), with no format change (the mark has been in the payload since v1). Three rules complete the design. Per-consumer horizons: RFC 0008’s replay delivers every surviving frame with its offset; the recovery driver suppresses per consumer — the Parquet path consumes only frames above the RFC 0008 checkpoint X (below it they are already published), the miner only frames above S (below it they are already folded into the snapshot; re-feeding would double-apply — the v1 hazard, now resolved by routing rather than by refusing to restore). The rule covers both orderings: in the steady state S ≥ X the miner consumes a suffix of what Parquet consumes; with a lagging snapshot (S < X) the miner additionally consumes the (S, X] frames the floor retained, closing its state gap while Parquet suppresses them. Truncation floor: the ingester passes the latest durable snapshot’s S to Wal::housekeeping as a retain floor, so the WAL never unlinks a frame no snapshot has captured (RFC 0008 §6.7 — closes the S < X template-drift hole, hazard #5). Stale-snapshot fallback: if recovery nevertheless finds the WAL truncated past S (external mutation — segment files manually unlinked from wal_root; the floor prevents the gap arising internally), it restores the snapshot, replays the surviving frames above S, and emits a structured warning naming the gap. The data side is complete provided the truncation did not exceed X — legitimate housekeeping never unlinks a frame above the checkpoint, so everything missing is in Parquet; manual deletion beyond X would be unrecoverable acknowledged-data loss, which is exactly why segment removal is reserved to housekeeping and never an operator action. Templates first seen inside the (S, X] gap may re-mint (drift surfaced via RFC 0010, not silent). New acceptance criteria: §3.5.3 (restore-equivalence), §3.5.4 (stale-snapshot fallback); the end-to-end driver contract is RFC 0008’s RFC0008.10.

Hot path. The per-tenant tree lives in process memory on the ingester. Tree operations (descend, simSeq, attach, widen) are hot-path; persistence does not happen synchronously per line.

Durability via WAL replay. [§3.4] (WAL-before-ack) means every line that reached the miner is in the WAL before the ingester acknowledged it. The tree state is therefore derivable from the WAL: a cold start with no snapshot replays the WAL in order through the miner and reconstructs the trees. This is correct but slow at scale; the snapshot mechanism is an optimisation on top.

Replay mode. Cold-start replay re-walks attach, widen, and expand_slot_types against the same code path live ingest uses. Doing so naively would re-fire every counter increment, every histogram observation, and every gauge update for the entire replay window, polluting steady-state metrics for the post-restart horizon (a 10-minute replay on a high-volume tenant could shift ourios.miner.merges by orders of magnitude in a few seconds). The miner therefore runs in replay mode until the WAL cursor reaches the live tip: domain events are processed and tree state is mutated exactly as in live ingest, but updates to the §6.8 metrics are suppressed (counters do not increment, histograms do not observe, gauges retain their previous value or, if the miner has never served live traffic, their zero / empty initialisation value). Suppressing the update path means the replay window contributes no data points — each instrument still surfaces on its first live measurement once replay completes (§3.1.2’s registry-defined set is satisfied by instrument registration plus real post-replay traffic, not by replay-window points). A single wal_replay_progress gauge (attribute ourios.tenant, value: fraction of the tenant’s replay window completed in [0.0, 1.0]) is exposed during replay so operators can see the cold-start curve and confirm replay finished. This metric is replay-only and is not part of the §3.1 mandatory set; it is documented here, not in §6.8’s table.

Snapshot mechanism. A snapshot is a rebuildable recovery-acceleration cache, not durable state. The WAL is the durable truth ([§3.4]); the snapshot exists only to shorten cold-start replay. A lost, absent, or corrupt snapshot is never a data-loss event — it degrades to a full WAL replay (the same path a miner that never wrote a snapshot takes). This framing is what licenses the store and recovery choices below: because the snapshot is cache, it may live on local disk, and a reader that cannot trust it may discard it without ceremony.

Target store: local disk (WAL-adjacent); object storage deferred. Snapshots are written to a local artefact next to the WAL (e.g. under the WAL root), not to object storage. [§3.6] makes local disk legitimate here precisely because the snapshot is cache, not truth — the constraint [§3.6] imposes is that no feature rely on local disk being durable beyond the WAL horizon, and snapshot recovery never does: anything the snapshot would have accelerated is still in the WAL. Object-storage snapshots are explicit future work (see §9); they would couple to the RFC 0009 §3.4 atomic-publish manifest to define a durable, multi-writer publish point, and are deferred for that reason.

Scope: per-tenant. One snapshot artefact per tenant tree, matching [§3.7]’s per-tenant trees. Recovery loads the latest snapshot per tenant independently; there is no cluster-wide combined artefact.

Cadence: per WAL-segment rotation. A snapshot is taken at WAL-segment-rotation boundaries. The snapshot records the WAL high-water mark — the WalOffset (RFC 0008 §6.1) up to which its tree state reflects appended frames — so that a future optimisation can resume replay from there rather than from the start of the log.

Format: a leading u8 version byte, then the payload. Byte 0 is the snapshot format version; the remaining bytes are that version’s serialised payload. The payload captures the per-tenant state needed to reconstruct the miner: the tree leaves (template token sequence, template_id, template_version, the (severity_number, scope_name) template key of §6.1, and the per-slot slot_types of §6.1), the structured-template-id map allocated in §6.2’s structured short-circuit, and the WAL high-water mark above. The concrete payload codec is an implementation detail behind the version byte — the version byte is what makes format evolution safe, so this RFC pins the framing, the captured state, and the rule that the reader dispatches on byte 0, and deliberately does not pin a specific serialisation codec.

Recovery algorithm. On ingester restart, per tenant:

1. Load the latest snapshot artefact for the tenant, if one exists.
2. If byte 0 is a known version: deserialise the payload, restore the tree, then replay only the WAL tail above the snapshot’s recorded high-water mark S — the driver delivers each replayed frame to the miner only when its offset is > S (per-consumer routing, RFC 0008 §6.6; active as of the 2026-06-12 v2 amendment above). If the surviving WAL no longer reaches back to S, apply the stale-snapshot fallback of the v2 amendment: restore, replay what survives, warn.
3. If byte 0 is an unknown version, or the snapshot is absent or corrupt: discard it and replay the full WAL via Wal::replay (RFC 0008 §6.1 API, RFC 0008 §6.6 recovery procedure), rebuilding the tree from scratch.

v1 scope — rebuild from a full replay; do not restore yet. (Superseded by the 2026-06-12 v2 amendment above — restore is now switched on; this paragraph is retained as the record of why v1 refused to restore.) The restore-then-replay-the-tail path in step (2) requires the RFC 0008 §6.7 checkpoint / replay-from-offset API (Wal::checkpoint and the CHECKPOINT sidecar), which was not yet implemented at the time. Restoring a tree from a snapshot and then replaying the full WAL (the only replay available without offset support) would double-apply every frame the snapshot already captured, corrupting the tree. So until offset-resume landed, recovery ignored the snapshot payload and rebuilt the tree from a full Wal::replay in both branches — the known-version branch did not restore. What landed in v1 was the snapshot format (the leading version byte and the recorded high-water mark) and the version-dispatch + WAL-fallback contract; v2 switches the restore path on with no format change, exactly as planned. v1 fully satisfied §3.5.1 (the artefact carries a leading version byte) and §3.5.2 (an unknown version is rejected and falls back to full WAL replay).

Snapshot-load telemetry. The wal_replay_progress gauge (above) remains the replay-only signal. A snapshot-load-outcome signal — distinguishing “snapshot restored,” “unknown version → full replay,” and “absent/corrupt → full replay” — is named here in prose; its concrete metric and attribute names go through the semconv weaver registry when the slice is implemented (§3.1.2), and are not invented as flat names in this RFC.

Migration. When the in-memory data model in §6.1 changes (new field, retired field, semantic change), the snapshot format’s version byte increments. Old snapshots are read-compatible only if the change is additive (new optional fields tolerated). For breaking changes, snapshots from the prior version are discarded and the tree is rebuilt from WAL replay. [§3.5]’s schema-change discipline applies: the change goes through an RFC.

7. Alternatives considered

Alternatives to Drain itself, evaluated as primary algorithms. Each is rejected for the reason given; some have a possible secondary role noted.

Spell (LCS-based online parser)

Spell uses longest-common-subsequence to compare a new line against existing templates. Per-line cost is O(template_count × line_length) without depth bounding, which is several orders of magnitude slower than Drain’s O(d) tree walk at the template counts we expect (10²–10⁴ per tenant). LCS also makes parameter positions ambiguous on lines where the same token recurs, because the LCS alignment can shift; Drain’s positional matching gives unambiguous parameter slots. Rejected as the primary algorithm.

IPLoM (iterative partitioning)

IPLoM does three passes over the entire log, each splitting clusters by a different criterion (token count, position, token uniqueness). This requires the full log up front and is offline by design. Rejected as the primary algorithm. Possible secondary role: a periodic offline reconciliation pass could use IPLoM to detect template fragmentation that Drain’s online structure missed (e.g. two leaves that should have been one because their discriminating token was spurious). This is a follow-up RFC topic, not a §6 commitment.

LenMa (length-based clustering)

LenMa groups lines by token-count length, then finds templates within each length group via a similarity-based second pass. The length-only initial grouping is close to Drain’s first level, but the absence of the token-prefix tree leads to more spurious merges within a length group (any two same-length lines are candidates, not just same-length-and-same-prefix lines). Drain’s tree is a strict refinement of LenMa’s grouping. Rejected as the primary algorithm — Drain dominates on the same workload.

LogPPT / LILAC / LLM-based parsers

Transformer-based parsers achieve higher accuracy on benchmark corpora (LogPAI scores) but require model inference per line. At the D1 hot-path budget (≥ 100k lines/s/core), per-line transformer inference is infeasible without specialised hardware that contradicts §1’s “single Rust binary” framing. Rejected as the primary algorithm. Possible secondary role: offline labeling-aid on the testdata/corpus/ to bootstrap a labeled set for confidence calibration; or as a periodic reconciliation pass similar to IPLoM. Both are deferred to follow-up RFCs.

Offline clustering (e.g. nightly hierarchical agglomerative)

Quality is high; latency is unacceptable. Logs ingested at 14:00 would not be queryable until the next clustering window completes. This contradicts §2’s online motivation. Rejected as the primary algorithm. Possible secondary role: the same reconciliation pass mentioned under IPLoM and LLM-based could use offline clustering to validate Drain’s online output and surface drift; a follow-up RFC if and when reconciliation becomes a real concern.

8. Testing strategy

Mapping to [§6.2]. Each technique below names the §5 scenarios it operationalises; the test code carries the matching id in a doc comment per docs/verification.md §2.3 so grep -R "H1.1" . resolves bidirectionally between RFC and tests.

• Unit tests for tree operations: tokenize, mask, descend, simSeq, widen, attach, build_params. Each operation tested in isolation against fabricated inputs. Covers: RFC0001.3 (tokenizer whitespace-only), RFC0001.4 (confidence ratio + decision boundary), RFC0001.7 (combined widening + type-expansion in one attach).

• proptest for §6.6 reconstruction: for every generated line shape (length, separator distribution, masking outcome), reconstruct(mine(line)) == line or mine(line).lossy_flag == true. Property failure blocks merge. Covers: H7.1, H7.4, §3.3.1.

• Corpus tests on testdata/corpus/ (fixed, anonymised; see docs/benchmarks.md §1): assert bounds on ourios.miner.template.count, ourios.miner.merges, reconstruction accuracy, parameter overflow rate. Regressions are build failures, not warnings. Covers: H1.1 (login/logout corpus arm), H7.1 (corpus arm).

• Confidence calibration test: on a labelled subset of the corpus, verify the three-zone classification in §6.3 against the human labels. Covers: H1.2.

• Merge-audit assertion (negative + positive): no widening or type-expansion completes without a matching audit event, and fresh-leaf creation does not emit one. Runs on every corpus pass and on the synthetic widening fixtures. Covers: H1.3, H5.1, H5.2, RFC0001.1 (negative — no event on creation), RFC0001.2 (rejection event for degenerate widening), RFC0001.7 (event ordering arm).

• Multi-tenant isolation (negative test): interleave lines from two synthetic tenants through a single MinerCluster; assert that templates mined under tenant A never appear in tenant B’s tree and vice versa. Implements docs/benchmarks.md E2. Covers: §3.7.1, §3.7.2.

• Per-ResourceLogs tenant derivation (miner-side stub): assert that when records carrying distinct derived tenant_ids arrive in the same ingest sequence, each lands in its derived tenant’s tree. The receiver-side test — that the wire-decode layer actually derives tenant_id per ResourceLogs.resource rather than per ExportLogsServiceRequest — is owned by RFC 0003 (see RFC 0003 §6.3); RFC 0001 owns only the miner-side contract. Covers: §3.7.3.

• OTLP-aligned template-key tests: hand-curated OtlpLogRecord fixtures exercising the §6.1 Template-key composition tuple. Assert that varying only severity_number produces distinct template_ids, varying only scope_name produces distinct template_ids, the severity_number = 0 (UNSPECIFIED) and scope_name = None edge buckets are each their own key value, and body.kind != AnyValue::String short-circuits per §6.2 step 0 with the §6.1 sentinel confidence = 1.0, lossy_flag = false. The time_unix_nano round-trip is a small unit test against the §6.1 record schema. Covers: H1.4, H1.5, RFC0001.9, RFC0001.10, RFC0001.11.

• Drift detection test: ingest a corpus where a template deliberately drifts mid-stream; assert that the drift query in §6.7 returns the drifted template within the expected window. Covers: H5.3.

• Crash recovery test (snapshot + WAL replay): SIGKILL the ingester between snapshot writes; assert that recovery reconstructs the same tree state that was acknowledged before the kill. Also corrupt the snapshot’s leading version byte and assert WAL fallback. This is [§3.4]’s crash-recovery test extended to cover the miner’s persistence layer. Covers: §3.5.1, §3.5.2.

• Restore-equivalence test: snapshot a tree at high-water mark S, append further frames, recover via restore-plus-tail-replay, and assert tree-state equality against a from-scratch control (the recovered state is compared field-by-field via the §6.9 snapshot payload of both trees, so the comparison itself can’t hide drift). A counter on the test sink asserts no frame ≤ S reached the miner. The stale-snapshot arm deletes the segments holding (S, tail]’s prefix, recovers, and asserts the structured warning names the gap while the surviving frames still fold. Covers: §3.5.3, §3.5.4 (the end-to-end driver half is RFC 0008’s RFC0008.10).

• Configuration tests: assert default values and the rejection of out-of-bounds settings at startup. Covers: §3.1.1 (default threshold = 0.7), §3.2.1 (default param byte limit = 256), §3.2.2 (limit > 1 KiB rejected).

• Metric collection test: assert the mandatory set equals the generated ourios_semconv::OURIOS_MINER_* constants (the registry is the source of truth), then construct a miner, ingest a small representative workload that exercises every instrument, collect its meter (global::meter("ourios.miner")) via an SDK in-memory reader, and assert the collected stream contains every §6.8 metric name (each appearing on its first real measurement, with the required attributes), with the instrument kinds and attributes in §6.8’s table, and that the ourios.miner.confidence.p50 / ourios.miner.confidence.p01 gauges track the same-attributed ourios.miner.confidence histogram quantiles. Covers: §3.1.2, RFC0001.8.

• Data-model contract tests: small unit tests against the template_id query semantics that RFC 0002’s DSL compiles to. These cover the cross-version vs. cross-alias distinction at the data-model layer; the DSL surface itself is tested in RFC 0002. Covers: RFC0001.5, RFC0001.6.

• Alias write-path tests: assert that an operator alias assertion emits a durable alias_asserted audit event under the §3.4 barrier and that folding the event log produces the expected per-tenant alias map; that resolves_to expands by the set (representative or member) and to {X} for an un-aliased id; that a retraction emits alias_retracted and drops membership on rebuild; and that an alias asserted in one tenant is invisible to another. The resolves_to DSL surface itself is exercised in RFC 0002 (RFC0002.9); these tests own the §6.7 write-path and per-tenant-map contract. Covers: RFC0001.12, RFC0001.13, RFC0001.14, RFC0001.15, RFC0001.16.

• Reader behaviour test: assert the §6.6 Reader render contract — for a lossy_flag = true row the reader returns the body bytes verbatim (no in-band prefix/marker) carrying Reconstruction::RetainedVerbatim, and does not call reconstruct(); for a faithful row it calls reconstruct() and carries Reconstruction::Faithful. The reader never silently substitutes one rendering for the other, and never mutates the body bytes to carry the marker. (The clean-path read-time template-registry lookup is out of scope of this scenario per §6.6.) Covers: H7.3.

• Overflow-path tests: synthesize a parameter exceeding the configured byte limit; assert the OVERFLOW marker, forced body retention, and metric increments. Wire the alert-rule fixture for the >1% rate trigger. Covers: H2.1, H2.2.

• Tokenizer-failure tests: feed lines with embedded NULs, malformed UTF-8, and over-cap lengths; assert the parse-failure path retains the body and sets lossy_flag = true. Covers: H7.2.

• Benchmark (criterion): per-line miner latency (target: median ≤ 10 µs/line on the §1 hardware baseline), ingest throughput (target: ≥ 100k lines/s/core, per docs/benchmarks.md D1). No §5 scenario; satisfies thesis-gate D1 directly at the Validated stage.

9. Open questions

Decisions explicitly deferred. Each must be resolved before this RFC’s status flips to accepted.

Persistence (from §6.9) — RESOLVED (2026-06-10). The three sub-questions are pinned in §6.9; the resolutions are recorded here and the remaining future work is the two items below them.

• Snapshot target store — RESOLVED: local disk (WAL-adjacent); object storage deferred. The snapshot is a rebuildable recovery-acceleration cache, not durable state (the WAL is the truth per [§3.4]), so it lives next to the WAL on local disk. [§3.6] permits this because recovery never relies on the snapshot surviving — a lost snapshot degrades to a full WAL replay. See §6.9.
• Snapshot cadence — RESOLVED: per WAL-segment rotation. A snapshot is taken at segment-rotation boundaries and records the WAL high-water mark (the WalOffset it was taken at). See §6.9.
• Snapshot scope — RESOLVED: per-tenant. One snapshot artefact per tenant tree, matching [§3.7]; recovery loads the latest snapshot per tenant. See §6.9.
• Object-storage snapshots (remaining future work). Pushing snapshots to object storage would couple to the RFC 0009 §3.4 atomic-publish manifest for a durable, multi-writer publish point; deferred until that line settles.
• Resume-from-high-water-mark replay — RESOLVED (2026-06-12): switched on as §6.9 v2. The RFC 0008 §6.7 checkpoint / offset-carrying-sink API is specified (RFC 0008’s same-day amendment); the known-version branch restores and replays only the tail above the snapshot’s high-water mark, with per-consumer routing, a housekeeping retain floor, and a stale-snapshot fallback. No format change was needed — exactly as this entry predicted. Acceptance: §3.5.3 / §3.5.4 + RFC0008.10. See the §6.9 v2 amendment.

Algorithm tuning (open until corpus exists).

• Floor default 0.4 — confirm against the corpus. If the lossy zone is too wide (many lines retained that “should have” been parse failures), tighten; if too narrow (too many parse failures on lines a human would accept), loosen. This is per-tenant tunable; the question is the out-of-the-box default.
• Tree depth d. Paper default 4; Drain3 default 4. Open question: do any of our representative corpora benefit from d = 3 or d = 5?
• Max children per node. Drain3 caps at 100; the cap acts as a safety against unbounded fan-out from a bad masking rule. Confirm 100 is right for our corpora, or motivate a different number.

Edge cases.

• Lines that contain a literal <*> (the wildcard sentinel we use in template strings) — escape on tokenize, or replace with a non-collision character (e.g. U+E000)?
• Multi-line log entries (stack traces). Paper assumes single-line. Ourios position: deferred to RFC TBD on the OTLP receiver, since multi-line reassembly happens before the miner sees the line.

Multi-tenancy and operational lifecycle.

• Tenant lifecycle. §3.7 commits to per-tenant trees but does not name when a tree is allocated (lazily on first ingest? eagerly via a control-plane command?), nor whether tenants can be paused, evicted under memory pressure, or deleted. Likely deferred to a future operator-console RFC, but the bookend events (TenantInitialised, TenantPaused, TenantDeleted) need to exist somewhere before §3.7 is operationally complete.
• Per-tenant fairness and back-pressure. A noisy tenant can monopolise WAL bandwidth, blow up the tree, and starve well-behaved tenants. RFC 0001 has no rate-limit or back-pressure event in scope; this overlaps with the OTLP receiver’s responsibility and likely lives in a future ourios-ingester RFC.
• Alias index creation mechanism (from §6.7) — RESOLVED (2026-06-07). Of the three candidates (operator-driven, automatic-inference, deferred entirely), the maintainer chose operator-driven + audited: an alias is an explicit, audited, reversible operator assertion on the §6.4 stream, never silently inferred (§3.1). The write path, the alias_asserted / alias_retracted events, the per-tenant alias-map projection the querier reads, and the eventual-consistency semantics are specified in §6.7 (“Alias write path”); the acceptance criteria are RFC0001.12– RFC0001.16 in §5.3. RFC 0002’s template_id.resolves_to(X) now has a defined backing index. Remaining future work: automatic inference is deferred to a possible propose → operator-confirm layer (proposals never enter the active index unconfirmed); see the §6.7 “Future work” note. The physical alias-map file/format and snapshot cadence are a storage decision owned by the RFC 0005 line, not RFC 0001 — RFC 0001 owns the model, the write path, and the criteria (sibling to the issue #147 split). Update 2026-06-12: the RFC 0005 line has made the v1 half of that decision — alias events persist as kinds 4–5 on the RFC 0005 §3.7 audit stream, and the querier derives the map by folding that stream at compile time (RFC 0005 §3.7.1; §6.7 amendment of the same date). The cached per-tenant artifact remains deferred behind the RFC 0009 §3.4 manifest fork.

Cross-RFC contracts pending.

• Querier ↔ live template registry (from §6.6). Reconstruction’s lookup(template_id, template_version) is called by ourios-querier, which runs in a separate process from the ingester that owns the live tree. Candidates: querier reads snapshots from object storage (eventually consistent with live), querier asks the ingester via RPC at query time (couples query latency to ingester health), or templates ride a separate Parquet side-stream alongside records (eventually consistent, no RPC, new data plane). RFC 0002 needs the answer before its DSL can compile; this RFC names the seam.
• Audit-event Parquet schema (from §6.4). The Rust audit struct is specified in §6.4; the on-disk Parquet column layout for template_audit belongs to a future ourios-parquet RFC. The §6.7 drift query assumes the schema exposes event_type, template_id, old_version, new_version, and timestamp as columns suitable for predicate pushdown.

Deferred to follow-up RFCs.

• Reconciliation pass (IPLoM / offline clustering / LLM-based labeling) — if real-world drift turns out to be more than §6.4’s online widening can handle, a periodic offline pass becomes interesting. RFC at that point.
• Cross-tenant template_fingerprint side column — only if a concrete consumer materialises (storage dedup across tenants, shared dashboards). Until then, do not add.

10. References

• He, P., Zhu, J., Zheng, Z., Lyu, M.R. “Drain: An Online Log Parsing Approach with Fixed Depth Tree.” ICWS 2017.
• Drain3: https://github.com/logpai/Drain3 (specific commit pinned in this RFC at the Specified-gate PR).
• LogPAI logparser benchmark: https://github.com/logpai/logparser
• CLAUDE.md §§ 2, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 4, 6.2, 6.3, 6.6.
• docs/hazards.md H1, H2, H5, H7.
• docs/benchmarks.md C1, C2, C3, C4, D1, E1, E2.
• docs/rfcs/0002-query-dsl.md §5.4 (template primitives in the DSL surface; required to expose drift detection).
• docs/verification.md §§ 2, 3, 6 (the maturity model and the acceptance-criteria contract this RFC will inherit at the Specified gate).
• Future: docs/architecture/miner.md (this RFC graduates there on acceptance).