Ourios

οὔριος — the fair wind that fills a ship’s sail.

Ourios is a log storage and query backend built on Apache Parquet, a Drain-derived online template miner, and Apache DataFusion. It is a work-in-progress; this book is where the design lives.

How this book is organised

• Architecture — the pillars, hazards, and shared vocabulary. The load-bearing reading for anyone new to the project.
• Benchmarks — the measurements that would falsify the thesis, stated as goals before any code exists to measure against.
• RFCs — design decisions in progress. Each RFC is a contract between the people working on Ourios about how a given subsystem will be built; once accepted and implemented, it graduates into an architecture document.
• Talks — lecture-length explanations of ideas from the RFCs, for when you want the background rather than the specification.

Project status

Greenfield. No code has been written yet; the design artefacts in this book are what exists today. Contributions, RFC discussion, and push-back on the invariants are all welcome — see CLAUDE.md in the repository root for the governing conventions.

Quickstart — single binary

The fastest path from zero to querying logs: one ourios-server process on your machine, local-disk storage, no auth. This is the development/evaluation posture — see Kubernetes (Helm) for the production topology and Authentication before exposing any listener beyond localhost.

1. Get the binary

Download a signed release archive (Linux; Apple-silicon and Intel macOS builds come from cargo build today):

curl -LO https://github.com/jensholdgaard/ourios/releases/latest/download/ourios-server-x86_64-unknown-linux-gnu.tar.xz
tar -xf ourios-server-x86_64-unknown-linux-gnu.tar.xz

Releases from v0.1.1 on attach offline provenance bundles (*.intoto.jsonl) alongside their assets, verifiable without any network round-trip:

gh attestation verify ourios-server-x86_64-unknown-linux-gnu.tar.xz \
  --repo jensholdgaard/ourios \
  --bundle ourios-server-x86_64-unknown-linux-gnu.intoto.jsonl

Or build from source with cargo build --release -p ourios-server.

2. Run it

The binary is one server with three roles — receiver (OTLP ingest), querier (the logs-DSL API), and the background compactor (on by default). Enable the two network roles and point everything at a scratch directory:

mkdir -p /tmp/ourios/data /tmp/ourios/wal

OURIOS_BUCKET_ROOT=/tmp/ourios/data \
OURIOS_WAL_ROOT=/tmp/ourios/wal \
OURIOS_RECEIVER_ENABLED=1 \
OURIOS_RECEIVER_GRPC_ADDR=127.0.0.1:4317 \
OURIOS_RECEIVER_HTTP_ADDR=127.0.0.1:4318 \
OURIOS_QUERIER_ENABLED=1 \
OURIOS_QUERIER_HTTP_ADDR=127.0.0.1:4319 \
./ourios-server

Startup prints the bound addresses and warns once that auth is in open mode:

receiver gRPC listening on 127.0.0.1:4317
receiver HTTP listening on 127.0.0.1:4318
querier HTTP listening on 127.0.0.1:4319

The ports are the OTLP defaults (4317 gRPC, 4318 HTTP) plus 4319 for the query API; the explicit 127.0.0.1 binds keep this quickstart on localhost (the server defaults to 0.0.0.0 for container use). Prefer a config file over env vars? See Configuration — --config ourios.yaml makes the file the sole source.

3. Send logs

Ourios speaks OTLP and nothing else — any OpenTelemetry SDK or Collector can ship to it unmodified. The tenant is derived from the service.name resource attribute.

With a Collector, point the OTLP exporter at it:

exporters:
  otlp:
    # host:port is version-proof for the gRPC exporter; recent
    # Collectors also accept scheme'd forms.
    endpoint: localhost:4317
    tls:
      insecure: true

Or hand-deliver one OTLP/JSON record for a first smoke test:

curl -s http://localhost:4318/v1/logs \
  -H 'Content-Type: application/json' \
  -d '{
    "resourceLogs": [{
      "resource": { "attributes": [
        { "key": "service.name", "value": { "stringValue": "checkout" } }
      ]},
      "scopeLogs": [{ "logRecords": [{
        "timeUnixNano": "1751971200000000000",
        "severityNumber": 9,
        "body": { "stringValue": "user 42 logged in" }
      }]}]
    }]
  }'

An empty {} response is the OTLP success shape. The batch is fsynced to the write-ahead log before that acknowledgement — kill the process mid-ingest and acknowledged data survives.

4. Query

POST /v1/query takes the logs DSL as plain text, with the tenant in a header:

curl -s http://localhost:4319/v1/query \
  -H 'X-Ourios-Tenant: checkout' \
  -H 'Content-Type: text/plain' \
  -d 'severity >= info | limit 10'

The response carries the total match count, the returned rows (bodies reconstructed from their mined templates), and scan statistics that show the Parquet pruning at work:

{
  "rows": 1,
  "stats": { "row_groups_scanned": 1, "row_groups_pruned": 0, "bytes_read": 4096 },
  "records": [ {
    "time_unix_nano": 1751971200000000000,
    "severity_number": 9,
    "body": { "kind": "rendered", "line": "user 42 logged in", "reconstruction": "faithful" },
    "...": "..."
  } ]
}

The DSL’s full grammar — field predicates, regex, time ranges, aggregation pipelines like service == "api" and severity >= error | count by template_id — is specified in RFC 0002.

Where to next

• Docker — the same server from the published image.
• Kubernetes (Helm) — the production topology on S3-compatible object storage.
• Authentication — static bearer tokens and OIDC; do this before any listener leaves localhost.
• The MCP surface (agents querying Ourios over the Model Context Protocol) rides the querier at /mcp — enable with OURIOS_QUERIER_MCP_ENABLED=1 (RFC 0027).

Configuration

Two mutually exclusive sources (RFC 0020 / RFC 0004):

• A YAML file via --config <path> — the file is then the sole source; the environment participates only through ${env:NAME} substitution inside it (with ${env:NAME:-default} defaults, $$ escaping — the OTel Collector data model).
• OURIOS_* environment variables when no --config is given — the container/dev posture.

Parsing is strict: a malformed value is a startup error in either mode, and in config-file mode an unknown YAML key is rejected too (unrecognised OURIOS_*-lookalike env vars are simply not read — there is no unknown-key concept in the environment).

A complete file example

storage:
  # local (a filesystem directory as the store — dev/single-node) or
  # s3 (object storage as the source of truth — production; RFC 0019).
  backend: s3
  s3:
    bucket: ourios-logs
    region: eu-central-1
    # Any S3-compatible provider: AWS, MinIO, R2, Ceph/RGW, …
    endpoint: https://s3.eu-central-1.amazonaws.com
    # Secret hygiene is enforced: credentials MUST be ${env:…}
    # references — inline literals fail startup.
    access_key_id: ${env:OURIOS_S3_ACCESS_KEY_ID}
    secret_access_key: ${env:OURIOS_S3_SECRET_ACCESS_KEY}
  # RFC 0022: per-key promoted attribute columns (service.name is
  # always promoted). Each key costs bytes on every row — opt in
  # deliberately.
  promoted_attributes:
    resource: [k8s.namespace.name]
    log: [http.request.method, http.route]

receiver:
  enabled: true
  grpc_addr: 0.0.0.0:4317
  http_addr: 0.0.0.0:4318
  # The WAL stays on local disk by design, S3 or not (RFC 0019).
  wal_root: /var/lib/ourios/wal

querier:
  enabled: true
  http_addr: 0.0.0.0:4319
  default_window_secs: 3600
  mcp:
    enabled: false

auth:
  # See the Authentication guide. Omit the whole section for open
  # mode (development only — the server warns once at startup).
  tokens:
    - name: edge-collector
      token: ${env:OURIOS_EDGE_TOKEN}
      tenants: [checkout, payments]
  oidc:
    issuer: https://dex.example.com
    audience: ourios-collector
    tenant_claim: groups
    name_claim: name

Environment variables (no --config)

Auth configuration is file-only — there are deliberately no OURIOS_AUTH_* variables; token values reach the file through ${env:…} references.

Docker

The release pipeline publishes a multi-arch image (amd64 + arm64) to GHCR, cosign-signed keyless:

docker pull ghcr.io/jensholdgaard/ourios:0.1.1

Verify the signature before trusting it — the identity is pinned to the exact release tag, so substitute both occurrences of the version when verifying another release (SECURITY.md is the authoritative verification reference):

cosign verify \
  --certificate-identity 'https://github.com/jensholdgaard/ourios/.github/workflows/image.yml@refs/tags/v0.1.1' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  ghcr.io/jensholdgaard/ourios:0.1.1

Image variants

Every release publishes three signed multi-arch images from the same source:

• default (:<version>) — glibc binary on distroless/cc.
• -static (:<version>-static) — static musl binary on distroless/static: no libc, libgcc, or libssl in the image, so the OS-package vulnerability surface scanners report is ~empty. Pick this one for the strictest supply-chain posture with the operational niceties (CA bundle, tzdata, nonroot passwd entry) kept.
• -scratch (:<version>-scratch) — the same musl binary on bare scratch, plus only the CA bundle TLS needs. Nothing else in the filesystem: the absolute minimum attack surface, but also zero operational conveniences — no tzdata, no passwd entry, and CA-bundle updates arrive only with Ourios releases rather than base-image bumps.

All three run identically (same flags, ports, and config surface below).

Run

Same binary, same configuration surface as the quickstart — env vars, or a mounted config file:

docker run --rm \
  -p 4317:4317 -p 4318:4318 -p 4319:4319 \
  -v ourios-data:/var/lib/ourios \
  -e OURIOS_BUCKET_ROOT=/var/lib/ourios/data \
  -e OURIOS_WAL_ROOT=/var/lib/ourios/wal \
  -e OURIOS_RECEIVER_ENABLED=1 \
  -e OURIOS_QUERIER_ENABLED=1 \
  ghcr.io/jensholdgaard/ourios:0.1.1

With a config file instead (the production posture — auth lives in the file):

docker run --rm \
  -p 4317:4317 -p 4318:4318 -p 4319:4319 \
  -v ourios-data:/var/lib/ourios \
  -v "$PWD/ourios.yaml:/etc/ourios/ourios.yaml:ro" \
  -e OURIOS_EDGE_TOKEN \
  -e OURIOS_S3_ACCESS_KEY_ID -e OURIOS_S3_SECRET_ACCESS_KEY \
  ghcr.io/jensholdgaard/ourios:0.1.1 \
  --config /etc/ourios/ourios.yaml

Secrets stay out of the file via ${env:…} references — pass through every variable your file references (the example forwards the auth token and, for an S3-backend file like the Configuration example, the store credentials; a local-backend file needs neither OURIOS_S3_* variable). The server shuts down gracefully on SIGTERM — docker stop flushes the ingest pipeline before exit.

Local note: any OCI runtime works — with containerd, nerdctl run/nerdctl compose take the same arguments.

Kubernetes (Helm)

The chart at deploy/helm/ourios deploys the production topology: three workloads, one binary, backed by S3-compatible object storage (AWS S3, MinIO, R2, Ceph/RGW, … — RFC 0019):

• receiver — a StatefulSet with a per-replica WAL PVC (the WAL is local by design; only data + audit go to S3);
• querier — a stateless Deployment, scales independently;
• compactor — a singleton Deployment.

Install

kubectl create secret generic ourios-s3 \
  --from-literal=OURIOS_S3_ACCESS_KEY_ID=… \
  --from-literal=OURIOS_S3_SECRET_ACCESS_KEY=…

helm install ourios deploy/helm/ourios \
  --set storage.backend=s3 \
  --set storage.s3.bucket=ourios-logs \
  --set storage.s3.region=eu-central-1 \
  --set storage.s3.endpoint=https://s3.eu-central-1.amazonaws.com \
  --set storage.s3.existingSecret=ourios-s3

(On AWS EKS, IRSA replaces the secret — leave existingSecret empty and annotate the service account with the role ARN; the two modes are mutually exclusive.)

The chart renders an RFC 0020 config file into a ConfigMap; credentials reach it as ${env:…} references resolved from the secret — never inline.

The chart’s README is the authoritative reference: full values.yaml documentation, the topology diagram, local-development (MinIO) recipes, and sizing notes. This page stays a pointer so the two never drift.

Sending and querying

In-cluster, point Collectors at the receiver Service (ourios-receiver:4317) and query the querier Service on 4319 — fronted by whatever ingress/TLS termination your cluster standardises on. Configure authentication before exposing either beyond the cluster boundary.

Authentication

Three postures, one enforcement path (RFC 0026 + RFC 0029). Whatever authenticates a request, the result is the same (name, tenants) binding: ingest batches must fall entirely inside the binding’s tenant set (whole-batch 403 otherwise, before the WAL), queries and MCP tool calls enforce the same set, and the name labels the audit trail and metrics. Rejections are deliberately undifferentiated — one 401 shape, no probing oracle.

Open mode (development only)

No auth section at all. Every request passes unbound; the server warns once at startup. Never expose an open-mode listener beyond localhost or a trusted network segment.

Static bearer tokens

The Collector-friendly baseline — static credentials in the config file, values injected via ${env:…} (inline literals fail startup):

auth:
  tokens:
    - name: edge-collector
      token: ${env:OURIOS_EDGE_TOKEN}
      tenants: [checkout, payments]   # or ["*"] for all tenants

Senders attach Authorization: Bearer <token>; with a Collector:

extensions:
  bearertokenauth:
    token: ${env:OURIOS_EDGE_TOKEN}
exporters:
  otlp:
    endpoint: ourios.example.com:4317   # TLS by default; gRPC host:port
    auth:
      authenticator: bearertokenauth

Comparison is constant-time; token values never appear in logs, errors, metrics, or audit events — only the name does.

OIDC (JWTs from an identity provider)

Adds standards-based machine identity in front of the same enforcement — any conforming issuer works; Dex (CNCF) is the recommended lightweight deployment and the one the acceptance suite runs against:

auth:
  oidc:
    issuer: https://dex.example.com
    audience: ourios-collector        # your client id
    tenant_claim: groups              # a string-list claim → the tenant set
    name_claim: name                  # the audit/metric label

Verification is local: the issuer is contacted once at startup (discovery + JWKS — an unreachable issuer fails startup, by design) and again only when an unseen key id appears (rotation). Signatures verify against the asymmetric allow-list only — RS256/384/512, PS256/384/512, ES256/384; alg: none and HMAC never verify.

Machine senders use the OAuth2 client-credentials flow — with a Collector this is zero custom code:

extensions:
  oauth2client:
    client_id: ourios-collector
    client_secret: ${env:DEX_CLIENT_SECRET}
    token_url: https://dex.example.com/token
    scopes: [openid, profile, groups]
exporters:
  otlp:
    endpoint: ourios.example.com:4317   # TLS by default; gRPC host:port
    auth:
      authenticator: oauth2client

Both halves coexist in one config — a static-token Collector and JWT-bearing senders authenticate side by side, each confined to its own tenant binding.

TLS

The listeners speak plaintext today; terminate TLS in front (ingress, service mesh, or an L4 proxy) — bearer tokens over plaintext are not auth. Native listener TLS is tracked on the auth epic.

OTLP log format — what crosses the wire vs. what Ourios consumes today

Status: investigation finding. Drafted 2026-05-13 to answer “is our template miner targeting the actual OTLP shape, or a made-up one?” Conclusion: the latter. This doc surfaces the gap and lists the RFC patches it implies; it does not change code.

The Ourios glossary commits the project’s ingest contract to OTLP over gRPC and HTTP — “we do not invent our own format” (docs/glossary.md, entry OTLP). The template-miner RFC (docs/rfcs/0001-template-miner.md) does not carry through on that commitment: §6.1’s record schema has eight fields, none of which exist on the OTLP wire, and the ingest signature today is MinerCluster::ingest(tenant_id, raw: &str) — a flat text line, not a structured LogRecord. This document closes the loop.

The first audience for this finding is the maintainer; the second is the RFC 0001 amendment PR and the future RFC 0003 (OTLP receiver) it implies.

1. What OTLP actually carries

The wire-level definition lives in opentelemetry-proto/opentelemetry/proto/logs/v1/logs.proto and the spec at opentelemetry.io/docs/specs/otel/logs/data-model. The relevant message hierarchy is:

LogsData
└── ResourceLogs[]
    ├── resource: Resource           ← Resource.attributes carries service.name, host.*, etc.
    ├── schema_url: string
    └── scope_logs: ScopeLogs[]
        ├── scope: InstrumentationScope   ← name, version, attributes
        ├── schema_url: string
        └── log_records: LogRecord[]

A single LogRecord carries:

Plus, inherited from the parent containers: the Resource attributes (the unit of “where did this come from” — typically service.name, host.name, k8s.pod.uid, etc.) and the InstrumentationScope name/version (which library/module emitted this record).

AnyValue is a oneof of: string_value, bool_value, int_value, double_value, array_value (recursive), kvlist_value (recursive map of strings → AnyValue), and bytes_value. The spec is explicit about the structured case:

Body MUST support AnyValue to preserve the semantics of structured logs emitted by the applications.

So a real OTLP emitter is at liberty to send a LogRecord whose body is, for example, {"msg": "user logged in", "user_id": 42, "from_ip": "10.0.0.1"} as a kvlist_value — with the parameters already structured out, not embedded in a free-text string.

2. What Ourios consumes today

MinerCluster::ingest(tenant_id: &TenantId, raw: &str) -> u64 (in crates/ourios-miner/src/cluster.rs). The pipeline:

1. tokenize(raw) splits on Unicode whitespace (crates/ourios-miner/src/tokenize.rs).
2. mask(tokens) runs UUID / IPv4 / NUM rules over the resulting &str slice (crates/ourios-miner/src/mask.rs).
3. descend + leaf lookup attaches to or creates a template (crates/ourios-miner/src/tree.rs).

The Parquet record promised by RFC 0001 §6.1 carries:

tenant_id, template_id, template_version, params,
separators, body?, confidence, lossy_flag

That’s the entire data model. Zero fields from the OTLP wire are reflected in the record.

3. The gaps

3.1 Severity is missing from the record (and from the template key)

severity_number is one of the most common operator query filters: “show me all ERROR-or-worse from service.name = api in the last hour.” Today the miner has no severity field.

Worse: the template key doesn’t include severity. A line emitted at INFO and the same line emitted at ERROR would currently collapse to one template_id. That’s a §3.1-class problem (“no silent template merges”) in disguise — two semantically distinct events sharing one id.

3.2 Timestamps are missing from the record

time_unix_nano and observed_time_unix_nano carry the data that the B1 thesis gate (“predicate-pushdown query latency on time/template/tenant filters”) explicitly measures. Without a time column we cannot run B1 at all.

Today there is no time field on the record. The Parquet writer PR (Phase 2 in docs/roadmap.md) cannot land without RFC 0001 §6.1 amending to add at least time_unix_nano.

3.3 Resource and scope are missing

Resource.attributes is OTLP’s “who sent this” partition key — in real deployments, service.name is the natural partition for template trees (it’s effectively the per-service template namespace). Today our tenant_id is operator-supplied and has no declared mapping from OTLP fields. We need to decide: tenant_id := resource.attributes["service.name"]? Or some configured mapping rule? RFC 0003 (OTLP receiver) is the place for this; RFC 0001 just needs to make resource_attributes a record column so the decision can land.

InstrumentationScope.name distinguishes the same body text emitted from different code paths in the same service. Likely also belongs in the template key — myapp.login and myapp.checkout emitting "request received" are different events.

3.4 Attributes carry the structured params we try to mine

In a structured-logging world, the values our mask() rules try to extract from text (NUMs, IPs, UUIDs) are typically already typed and separated by the SDK as Attributes. A modern emitter sends:

• body = "user logged in"
• attributes = {"user.id": 42, "client.address": "10.0.0.1"}

Not:

• body = "user 42 logged in from 10.0.0.1"
• attributes = {}

Our miner gets the second form and does work to reconstruct roughly what the first form already had. Worse, given the first form, we currently mine "user logged in" as a flat fixed template and lose the typed attribute values entirely — they’d never reach the Parquet record. The operator query “show me all logins from client.address = 10.0.0.1” returns nothing.

The implication for the miner is significant: the params slot on the record cannot be only “things mask() extracted from the body string.” It must also carry the OTLP attributes of the record — either as a sibling column (operator-queryable) or folded into the existing params shape (more complex).

3.5 Body is not always a string

AnyValue body. Today ingest(raw: &str) cannot accept a structured body at all. Three plausible paths:

1. Render-to-string at the receiver. Convert structured Body to a canonical JSON-ish string before handing to the miner. Loses the structure but preserves the existing miner shape. Risk: §3.3 (“bit-identical body reconstruction”) requires the rendered form to round-trip; canonicalising arbitrary AnyValue trees is non-trivial.
2. Treat structured Body as not-mineable. Store it verbatim in the body? column with lossy_flag = false (it’s an explicit structured value, not a lossy reconstruction); the miner emits a template_id of “structured body” and the query path knows to read body? directly. Simpler, gives up templating for those records.
3. Mine inner string fields. If body is a kvlist_value with a "msg" field, mine msg as the line. Pragmatic but ad-hoc; the field name is convention not spec.

Path (2) is the cleanest minimum; path (1) is the eventual ambition; path (3) is a configurable convenience layer. All three need an explicit spec decision.

3.6 Trace correlation is missing

trace_id, span_id, flags are how operators correlate logs to spans in the same trace. Real operators use this constantly. Today: no fields, no support. Add as record columns.

3.7 The ingest signature itself is wrong

ingest(tenant_id: &TenantId, raw: &str) cannot accept any of the above. The eventual signature is roughly:

#![allow(unused)]
fn main() {
fn ingest(&mut self, record: &OtlpLogRecord) -> u64
}

…where OtlpLogRecord is a struct that mirrors the OTLP wire shape (or borrows directly from a tonic-decoded protobuf message). This is a breaking change to the cluster’s public surface and is rightly the territory of RFC 0001’s amendment.

4. Implications

4.1 RFC 0001 §6.1 needs amendment

The minimum schema additions to make the record OTLP-faithful:

Plus an explicit decision on:

• Template key. Is the leaf identified by (masked_body_tokens) alone, or by some tuple of (severity_number, scope_name, masked_body_tokens)?
• body representation. AnyValue → what does the miner see? (Per §3.5 above.)
• tenant_id derivation. What OTLP field(s) define it?

4.2 RFC 0001 §6.2 (algorithm) needs a tokenize/mask amendment

tokenize + mask are designed for text. Once Body is AnyValue, the front of the pipeline branches: structured Body skips the tokenize/mask path entirely (or uses path (3) above on a configured field). The algorithm spec needs to acknowledge this fork.

4.3 RFC 0003 (OTLP receiver) becomes a prerequisite, not a follow-up

Today’s roadmap.md §5 lists the OTLP receiver as “first post-MVP shipping PR series.” That sequencing assumes the receiver is just the wire-decode-and-forward layer for an already-OTel-aligned record schema. With the gaps in §3 above, the receiver and the schema co-evolve: you cannot define the record without knowing what the receiver hands you, and you cannot define the receiver without knowing what the record expects. RFC 0003 should be drafted alongside the RFC 0001 amendment, not after it.

4.4 The Phase-3 corpus + bench need an OTLP-shaped corpus

The corpus runner (ourios-bench, Phase 3) cannot validly exercise the C2 thesis gate (template-count convergence) on flat-text input if the production input is OTLP. The corpus input must itself be OTLP-shaped — either a pre-recorded batch of LogsData protobuf, or a generator that emits realistic LogRecords including the structured-Body and attributes-bearing variants.

4.5 The current cluster’s behaviour is not fully wrong, just narrow

Plain-text traditional logs (Syslog, Log4j, slog with default text formatter) produce LogRecords with string Body and near-empty Attributes. The current miner handles those records correctly modulo the missing timestamp / severity / resource columns. So the current code is not throw-away; it’s the text arm of a fork that the OTLP-aware ingest will need.

5. Recommendation

Three follow-ups, in order:

1. Patch RFC 0001 §6.1 + §6.2 (a meta:-shaped change to the record schema and the algorithm spec). Land the new columns, the template-key decision, and the AnyValue handling fork. Do this first because the rest of the work depends on it.

2. Draft RFC 0003 — OTLP receiver. Cover (a) the wire-decode layer (tonic for gRPC, axum/hyper for HTTP/protobuf, against the official opentelemetry-proto crate); (b) the OtlpLogRecord → MinerCluster mapping; (c) the tenant_id derivation rule; (d) the WAL-before-ack sequencing under the new structured shape (§3.4); (e) build-vs-depend evaluation (tonic + hand-roll vs. embedding the rotel Rust collector vs. running the OTel Collector out-of-process and forwarding).

3. Patch the miner crates to consume the new record shape and route Body through the AnyValue fork. Update the roadmap to reflect that OTel-native ingest is no longer strictly post-MVP for the C2 gate’s validity.

The user-visible effect: the eventual benchmarks measure what an actual OTel deployment would experience, not a flat-text caricature of it. The thesis claim of “Parquet + template mining

• DataFusion is the right stack for OTel logs“ becomes testable in the form an operator would actually evaluate it.

6. References

• OTLP logs.proto: github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto
• OTLP common.proto (AnyValue, KeyValue): github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/common/v1/common.proto
• OpenTelemetry Logs Data Model spec: opentelemetry.io/docs/specs/otel/logs/data-model
• RFC 0001 §6.1 (current record schema): docs/rfcs/0001-template-miner.md
• Glossary entry OTLP (the load-bearing commitment): docs/glossary.md

Last updated: 2026-05-13.

Hazards

Referenced from CLAUDE.md §4 (“Before any change to the hot path, re-read docs/hazards.md”) and §10 (“When in doubt: 1. Read docs/hazards.md”). This document is the load-bearing reading for any hot-path reviewer. Each hazard names a specific failure mode, the mitigation we have committed to, the detection signal, and the rule for when a deviation is a tuning question vs. an architectural one.

How to use this document

• Before opening a PR that touches any subsystem named in a hazard section: re-read that section. The PR description must explicitly say which hazard it touches and how the change preserves the mitigation.
• In review: if a hazard is touched and not addressed in the PR description, that is a block, not a nit.
• In production: the named detection signals are the alerts that cannot be silenced without an RFC. They exist precisely so the failure mode is visible before it corrupts data.

Hazards map onto invariants in CLAUDE.md §3. Hazards describe what goes wrong; invariants describe what we promised. They are two faces of the same constraint.

H1 — Template miner correctness

Failure mode. The miner merges semantically-distinct templates because they share token structure. The canonical horror: user logged in <*> and user logged out <*> differ in one token; below a permissive threshold they merge into user logged <*> <*>. A query for the login event silently returns logout rows. The operator never knows.

Mitigation.

• Default similarity threshold ≥ 0.7 (strict).
• Lowering the threshold below 0.7 requires an RFC, not a config change.
• Three-zone confidence model: clean match (≥ threshold) / lossy match (floor ≤ x < threshold, retain body — reconstruction still succeeds, so lossy_flag is not set) / parse failure (< floor, retain body, increment counter). lossy_flag is reserved for the H7 case (genuine tokenizer / preprocessing failure where reconstruct(record) != ingested_bytes is possible); it is not a low-confidence signal. See docs/rfcs/0001-template-miner.md §6.6 for the precise definition.
• Every template-widening event is audited: the audit record names the old template, the new template, tenant, timestamp, and reason.

Detection. All metrics carry tenant_id; some carry service.

• merges_total counter: spike on stable input → service-version change or threshold drift.
• body_retention_ratio gauge: rising → input shifted or threshold is too tight.
• confidence_p01 histogram tail: collapsing → many matches are barely passing; threshold should be revisited.
• parse_failures_total: nonzero is genuine failure, not lossy.

Escalation. A spike on one tenant is a tuning question (masking rules, per-tenant threshold). A spike across many tenants on a stable corpus is a policy question — RFC.

See also. CLAUDE.md §3.1; docs/rfcs/0001-template-miner.md §§6.3–6.4; docs/benchmarks.md C2 (template count convergence), C3 (merge rate).

H2 — Parameter cardinality blowup

Failure mode. A params slot captures something it should not — an entire stack trace, a base64 payload, a request body, a megabyte JSON blob. Parquet’s dictionary encoding for that column collapses (every value distinct). File sizes explode. Query latency on that column degrades by orders of magnitude. The backend’s compression claim evaporates for that workload.

Mitigation.

• Per-parameter byte limit, default 256 B, ceiling 1 KiB — raising the ceiling requires an RFC.
• Overflow spills the original value into the body column; the params slot is replaced by a truncation marker (length + hash, no original payload).
• Counter increments on overflow.

Detection.

• params_overflow_ratio per service: alert when > 1 % of lines for any one service hit overflow.
• Parquet column-size variance: a column whose dictionary efficiency drops sharply between compactions usually means a new overflow pattern.

Escalation. Service-specific spike → masking rule that pre-redacts the offending field. Broad spike → revisit the limit (still ≤ 1 KiB). Anyone proposing > 1 KiB → RFC.

See also. CLAUDE.md §3.2; RFC 0001 §6.5; benchmarks C4.

H3 — WAL durability vs. latency

Failure mode. The ingester acknowledges an OTLP batch before the write is durably persisted. The ingester then crashes (process kill, host failure, container reschedule). The producer believes the data was accepted; we have lost data we promised to keep.

Mitigation.

• An ack is emitted only after fsync (or equivalent durability primitive) on the WAL.
• Batched fsync with an explicit operator-tunable knob: default flush every 100 ms or when the current segment fills, whichever first.
• Crash-recovery test is part of CI: SIGKILL the ingester mid-batch, restart, assert no acknowledged data is missing. Test runs on every PR; failure blocks merge.
• Replication, when added, is in addition to the WAL, not a replacement.

Detection.

• ingest_ack_latency_p99: rising trend usually means fsync is the bottleneck.
• wal_unflushed_bytes: bytes acked but not yet on durable storage — must always be bounded.
• CI crash-recovery test: any failure is critical, regardless of flake history.

Escalation. Fsync latency rising → tune batch size or move to faster storage. Ack-without-fsync ever observed in code review → P0 bug, hotfix path.

See also. CLAUDE.md §3.4; RFC 0008 (WAL design); benchmark D2 (compaction keeps up).

H4 — The small-file problem

Failure mode. WAL segments get rotated and flushed to Parquet too eagerly. The result is thousands of small files per tenant per day. Object-storage LIST calls dominate query planning time. Cold cache hits are murderous. Operators see “query took 12 s on 4 GB of logs” and lose faith in the backend.

Mitigation.

• Target row-group size 128 MB – 1 GB inside each Parquet file.
• Target file size 256 MB – 2 GB post-compaction.
• Background compaction job per tenant; cadence is a tunable.
• Compaction is required to keep the WAL backlog bounded under sustained ingest (D2).

Detection.

• File-size histogram per tenant: fewer than 5 % of files below 128 MiB at steady state.
• File count vs. data volume: file count must grow sub-linearly with bytes ingested.

Escalation. Skewed file-size distribution on a single tenant → compaction tuning. Sustained small-file emission across the cluster → ingest-scaling block, RFC.

See also. CLAUDE.md §4 hazard 4; benchmarks D3.

H5 — Template schema evolution across deploys

Failure mode. A service ships a new version. Log format changes — a new field, a renamed token, reordered words. The template tree built from last month’s logs no longer matches the new format cleanly. Queries against template_id = X start returning incomplete results because some rows are now stored under template_id = X'. The operator sees a 30 % drop in event volume and misdiagnoses it as an outage.

Mitigation.

• Templates are versioned: a template’s internal representation can change; the logical identity persists across versions.
• Explicit alias mechanism: template_id.resolves_to(X) in the DSL resolves a query across all aliases of X.
• Drift detection is a first-class query — operators can ask “what templates drifted in the last 24 h?” and get a list.
• A new template_version emits an audit event, just like a merge.

Detection.

• Spike in distinct template count immediately after a deploy → expected; investigate only if it persists past the deploy window.
• Diff between template_id = X and template_id.resolves_to(X) result counts → measures alias coverage.
• Audit event volume: drift events should correlate with deploy cadence, not appear randomly.

Escalation. Alias graph becomes a tangle (templates with > N aliases or cycles) → revisit alias semantics, RFC. Drift correlated with deploys → expected; not an alert.

See also. CLAUDE.md §3.5; RFC 0001 §6.7.

H6 — Query DSL vs. DataFusion SQL surface

Failure mode. A user-facing query surface accidentally exposes DataFusion specifics — a SQL keyword leaks into an error message, a planner hint becomes documented, a join type that doesn’t make sense in a logs context becomes reachable. We then cannot upgrade DataFusion or change the planner without breaking saved user queries and dashboards. The DSL has become a contract we never intended to sign.

Mitigation.

• The DSL is a separately specified layer (docs/rfcs/0002).
• All DSL constructs compile to DataFusion LogicalPlan, never to SQL strings. SQL never appears in any user-visible output.
• No SQL escape hatch by default. If one is added later, it ships under a separate RFC, sandboxed, opt-in, and tenant-gated.
• DSL evolution is a written semver contract with users; major versions ship with a deprecation window.

Detection.

• PR review: any test or error message containing the substring “DataFusion” or referring to a DataFusion type by name in a user-facing surface is a block.
• Any code path that constructs SQL strings from user input is a block.
• User report: “this query worked yesterday after the upgrade” triggers a regression review.

Escalation. Leak found in user-facing surface → block + hotfix. Recurring temptation in implementation → tighten the API boundary, move shared helpers behind a non-exported module.

See also. CLAUDE.md §4 hazard 6; RFC 0002.

H7 — Bit-identical body reconstruction

Failure mode. An operator opens the UI and asks “show me what was actually logged.” We render the row from template + params and produce a string that drops a space, a quote, a separator, or a trailing newline. The operator chases a bug that doesn’t exist — or, worse, fails to chase a bug that does, because the rendered line looked normal.

Mitigation.

• The miner either captures inter-token whitespace and separators or it sets lossy_flag = true on the row. There is no third option.
• Reconstruction is a property test against the testdata corpus: for every non-lossy row, reconstruct(record) == ingested_bytes exactly. Property failure blocks merge.
• The reader honours lossy_flag. The UI surfaces lossy rows with an explicit warning (“this row’s body cannot be exactly reconstructed”) rather than rendering them.
• Tenants may opt into default-on body retention at a storage cost.

Detection.

• Reconstruction property test (CI): zero failures, ever, on the committed corpus.
• body_retention_ratio gauge: a sudden rise indicates input distribution change OR a regression in whitespace capture.
• User complaint of “the rendered log does not match what we sent” → reproduce, add to corpus, fix.

Escalation. Ever fails on a real-world corpus → block + hotfix. Whitespace-capture state machine becomes a complexity sink → simplify by retaining more bodies; the storage cost is real but acceptable, lying to the user is not.

See also. CLAUDE.md §3.3; RFC 0001 §6.6; benchmarks C1.

H8 — Replication-induced dedup under clock drift

Forward-looking hazard. Ourios does not currently replicate at ingest, so this hazard is dormant. It is recorded here so that any future RFC proposing replication starts with the failure mode already understood.

Failure mode. A multi-ingester replication design quietly introduces a storage multiplier if dedup is keyed on filename and a time window. Even sub-second clock drift between replicas causes the dedup pass to miss duplicates: each replica writes “the same” record under a slightly different filename or window, the dedup miss is invisible to any single replica’s logs, and the user pays for redundancy they thought they had bought once. A widely-deployed log backend was found in 2026 to be carrying a ~2.3× storage multiplier from exactly this failure mode, motivating a full re-architecture of its durability layer.

Mitigation.

• Currently: not a concern. Ourios does not replicate at ingest. Per CLAUDE.md §3.4, durability is per-ingester fsync on the WAL; per §3.6, object storage is the long-term truth. Replication, if introduced later, is “in addition to the WAL, not instead of it.”
• When replication is proposed: dedup MUST be content-keyed, never time-windowed. The producer (or the OTLP layer) supplies an idempotency key — a content hash combined with a producer identifier and a sequence number. Ingesters treat the key as opaque. Clock drift becomes irrelevant to dedup correctness.
• Time-window dedup is rejected by default, regardless of how cheap or convenient it appears. An RFC proposing it must read this hazard and address it explicitly.

Detection.

• bytes_stored / bytes_received per tenant: must hover near 1 on a single-replica deployment. A multiplier > 1 + ε on a single-replica deployment indicates double-writing somewhere upstream.
• On a future replicated deployment: this ratio should remain near 1 after dedup; a sudden rise indicates clocks drifted and the dedup pass is missing duplicates.
• A “dedup hit rate” metric on the dedup pass — a sudden drop signals that something is masking what should be visible duplicates.

Escalation. Replication proposed → RFC must explicitly address this hazard. Time-window dedup proposed → block, redirect to a content-keyed approach. Storage multiplier > 1.05 on a single- replica deployment → P0 investigation, something is double-writing.

See also. CLAUDE.md §3.4 (WAL durability), §3.6 (object storage as truth). Cautionary tale: Grafana Loki’s 2026 re-architecture replacing replicate-at-ingest with Kafka-as-durability (InfoQ news, April 2026).

Adding a new hazard

A new hazard belongs in this document if all of the following hold:

• It is a failure mode that silently corrupts data, lies to the user, or destroys the project’s value proposition.
• It is not obvious from reading the code (otherwise it is a bug, not a hazard).
• It has at least one named mitigation in the codebase or a committed RFC.

A new hazard is added via a meta: RFC, the same path as changes to CLAUDE.md §3 invariants.

Verification

Status: active. This document is the process spec. The proposed amendments to docs/rfcs/README.md and CLAUDE.md at the bottom of this file are tracked separately and applied in their own PR once the structure here is settled.

What this doc is for

The Ourios docs already define invariants (CLAUDE.md §3), hazards (docs/hazards.md), per-RFC testing strategies (RFC §6 — see §2.5), thesis-gates (docs/benchmarks.md), and the project’s testing discipline (CLAUDE.md §6.2). What is missing is the described process that connects them: how a contributor (human or agent) takes a §3 invariant or an H-x hazard, turns it into reviewable acceptance criteria, turns those into red tests, drives them green, and gates an RFC’s transition to accepted.

This doc fills that gap. It is human-readable process, not test code, not tooling, not a coverage policy.

1. The flow

Six links, four gates between them. The diagram below names them; the text after walks the chain in the order a contributor encounters it.

Invariant (§3)        Hazard (H-x)        RFC (§§1–4)
       \                 |                   /
        \________________|__________________/
                         ↓
                Acceptance criteria
              (RFC §5 — normative,
               structured prose)
                         ↓
                   Red tests
                  (compile, fail)
                         ↓
                  Green tests
              (unit + property + corpus)
                         ↓
                  Validated
        (corpus + thesis-gates pass on
         representative inputs)

The chain has two entry points (Invariant, Hazard) that converge on the third (RFC). An RFC enumerates the invariants and hazards it touches in its §1 Summary; reviewers verify the enumeration is exhaustive at the Drafted → Specified gate.

Invariant → RFC. A CLAUDE.md §3 invariant is a project-level promise. Until an RFC operationalises it, the invariant is a known debt. §4 Entry points describes the three doors into this chain.

Hazard → RFC. Each hazards.md H-x item names the RFCs and crates responsible in its Mitigation and See also fields. The hazard does not move; the RFC inherits the obligation to defend it.

RFC → Acceptance criteria. Acceptance criteria live in RFC §5 (see §2) and translate the invariants and hazards the RFC touches into testable scenarios. The Specified gate ratifies the list.

Acceptance criteria → Red tests. A red test is a compiling stub that fails — typically with todo!() or unimplemented!() — and references the scenario id in a doc comment. Red tests are not required at the Specified gate; they are the artefact of crossing the Red gate, immediately before implementation begins. Forcing stubs to compile at Specified would push authors into premature specificity about types and signatures. Red stubs are tagged #[ignore] so the outer CI loop stays green while the inner loop (the implementor running cargo test -- --ignored locally) sees the todo!()s fire as a TODO list — see §3 for the two-loop spec.

Red tests → Green tests. Implementation lands; each stub becomes a real test that passes; unit, property, and corpus tests cover the scenario as CLAUDE.md §6.2 dictates. The Green gate confirms every §5 acceptance criterion has a matching passing test.

Green tests → Validated. The thesis-gates in benchmarks.md §7 that the RFC’s pillars touch must pass on representative corpora. Once they do, the RFC’s status: flips to validated. Maintainer sign-off then flips it to accepted.

2. Acceptance criteria

The contract here is single-typed: every invariant or hazard the RFC touches resolves to one or more scenarios, each with an id, a leading clause grammar, and a greppable counterpart in test code.

2.1 Format

Structured prose using bold leading clauses. Each scenario carries a short numeric id (see §2.2) and follows the Given / When / Then / And pattern:

Scenario H1.1 — Semantically distinct templates do not silently merge

• Given a corpus containing user logged in <*> and user logged out <*>
• When similarity threshold is 0.7 (default)
• Then the two remain distinct template_ids
• And any widening produces an audit event recording both old and new templates

The format is the markdown the project already uses, not Gherkin. We do not adopt .feature files, cucumber-rs, or any other BDD tooling: the test code is Rust (CLAUDE.md §6.2), and the scenario lives in the RFC where reviewers are already reading. A second source of truth — a .feature file checked separately — would drift, and the tooling does not pay for itself at our scale.

2.2 Scenario ids

Three id grammars, chosen to make the source of the obligation visible at a glance:

• H<n>.<m> — hazard-rooted; H1.1 is the first scenario defending hazard H1.
• §3.<n>.<m> — invariant-rooted; §3.4.2 is the second scenario defending CLAUDE.md §3.4 (WAL-before-ack).
• RFC<NNNN>.<m> — RFC-internal; reserved for scenarios that defend an RFC’s own design decisions, not a numbered invariant or hazard. Example: RFC0001.3 for a Drain3-extension behaviour that is not load-bearing for any §3 invariant but is part of the RFC’s contract.

Numbers within an id family are assigned in the order scenarios are written and never renumbered. A retired scenario keeps its number; new scenarios append. This gives git log -S "H1.1" a stable target across the lifetime of the project.

2.3 Greppability

The id is referenced from the test code in a doc comment, exactly:

#![allow(unused)]
fn main() {
/// Scenario H1.1 — Semantically distinct templates do not silently merge.
/// See `docs/rfcs/0001-template-miner.md` §5.
#[test]
fn login_and_logout_do_not_merge_at_default_threshold() { /* … */ }
}

grep -R "H1.1" . then yields the scenario in the RFC, the test in the crate, and any cross-references in the docs — bidirectional in one command. If a scenario is renamed, both ends move in the same commit.

2.4 Normative vs. exhaustive

Acceptance criteria are the normative tests an RFC promises will exist. The implementation will write many more — regression tests, edge cases, performance smokes — and those are not catalogued in the RFC. Reviewers ratify the normative set: every invariant and hazard the RFC touches has at least one scenario, and the scenarios as written are testable in principle.

The opposite mistake — listing every test the implementation will ever write — turns the RFC into a test plan and freezes the implementation. We do not do that.

2.5 Location in the RFC

Acceptance criteria are a new RFC §5, immediately before Testing strategy. The placement is deliberate: criteria are the spec the testing strategy operationalises, so reviewers reading the RFC top to bottom encounter the what before the how. The proposed amendment to docs/rfcs/README.md at the bottom of this file captures the renumbering: existing §5 Testing strategy shifts to §6, Open questions to §7, References to §8.

3. The RFC maturity model

Five stages, four gates. Each stage is a value of the RFC’s status: frontmatter field, so an RFC’s current maturity is visible without reading the body:

accepted is a distinct terminal status — it represents maintainer sign-off after Validated is reached. rejected and superseded are the other terminals, all three reachable from anywhere in the maturity ladder. A Drafted or Specified RFC may be rejected on review without ever reaching Red; an Accepted RFC may be superseded by a later one without re-traversing the chain.

The table is the spec; the paragraphs below explain what artefacts exist at each stage and what a reviewer is ratifying.

Drafted. The RFC has §§1–4 (Summary, Motivation, Proposed design, Alternatives considered) plus §§7–8 (Open questions, References) filled enough that two engineers reading it would produce roughly the same implementation. Acceptance criteria (§5) and Testing strategy (§6) may be empty or stubbed. The PR is open with status: drafted; review focuses on whether the design is correct in principle. The gate to Specified is a peer reviewer saying “yes, this design is what we want — now write down the contract.”

Specified. §5 Acceptance criteria is filled. Every invariant in CLAUDE.md §3 and every hazard in hazards.md that the RFC touches has at least one numbered scenario. §6 Testing strategy references those scenarios and names the technique (proptest, corpus, criterion) for each. The reviewer asks one question: could a competent implementor turn each criterion into a test as written? If the answer is no — for any criterion — the RFC has a gap and goes back to Drafted.

The Specified gate is the most valuable. It is the only gate where the cost of being wrong is bounded by review time rather than implementation time. We do not require test stubs to compile here; forcing stubs would push authors into premature decisions about function signatures, traits, and module structure, which is the Red gate’s job, not this one.

Red. Test stubs exist, are tagged #[ignore], and fail when run. Each stub carries a doc comment naming its scenario id (§2.3). Stubs may be todo!(), unimplemented!(), assert!(false) — anything that compiles and fails. Implementation may begin.

The Red signal lives at two granularities, deliberately:

• Inner loop (local dev cycle). The implementor working on a stub runs cargo test <name> -- --ignored and watches the todo!() panic. Each panic is one TODO item; as the body fills in, the #[ignore] comes off and the test joins the default run.
• Outer loop (CI). Default cargo test skips ignored tests, so the Red-stage PR lands cleanly through branch protection rather than fighting it. CI’s signal that the Red gate is satisfied is structural: stubs compile, every §5 scenario has an #[ignore]’d test with a matching id, and cargo test -- --include-ignored exits non-zero on each. (The greppability contract in §2.3 makes the per-scenario coverage check mechanical — grep -R "H1.1" returning both the RFC line and the test stub line is the assertion.)

The two-loop split is what lets us treat the Red status as a landable, mergeable state rather than a half-broken branch. A Red-stage main is healthy: outer loop green, inner loop fully populated with the work that needs doing.

The gate is mechanical: every scenario in §5 has at least one stub with a matching id, the stub is tagged #[ignore], and cargo test -- --include-ignored exits non-zero on each.

Green. Implementation lands. Every stub becomes a real test; unit, property, and corpus tests cover their scenarios as CLAUDE.md §6.2 dictates. cargo test --all-features passes. The reviewer confirms each §5 criterion now resolves to a passing test (the greppability contract makes this mechanical). No performance claim is made yet.

Validated. Every thesis-gate in benchmarks.md §7 that the RFC’s pillars touch passes on representative corpora. Maintainer inspects the corpus and the delta against target, signs off, and flips status: to accepted. The RFC is now binding; subsequent changes go through the regression handling in §3.1.

3.1 Regression handling after Validated

A failing test on a previously-Validated RFC is, by default, the test doing its job. The RFC does not reopen. Standard PR workflow: fix the regression, ship the patch, the test stays green.

The RFC reopens only when a single criterion fails repeatedly on the same code path — concretely, when the same scenario id fails on three independent commits within a 30-day rolling window, or when two distinct regressions touch the same criterion within the same window. The threshold is that the criterion has stopped being a defence and has become a moving target; that is a signal the RFC’s commitment is under-defended or under-specified, and the design (not just the implementation) needs revisiting.

This threshold is informal at the Specified gate; it sharpens once real signals exist. The point of writing it down now is that contributors do not race to reopen RFCs on every CI flake, nor pretend a repeated structural failure is just bad luck.

Thesis-gate failures during Validated follow benchmarks.md §7’s existing escalation rule (one fail on one corpus → tuning RFC; two or more → pillar RFC, pause), not this section.

3.2 Outer loop vs. inner loop

The maturity model is the outer loop. Each stage names a checkpoint that an external reviewer can verify: at Specified the scenarios are written, at Red the stubs compile and fail, at Green the same stubs pass. Nothing in the outer loop says how a developer fills the Red → Green transition.

The recommended inner loop is classic Beck-style TDD: write one failing test, make it pass with minimal code, refactor, triangulate by writing the next test that forces generalisation, repeat. It is not mandatory — a developer who prefers to stub all scenarios up front and implement against them is welcome to. The outer loop only requires that every §5 scenario has a stub by the Red gate and a passing test by the Green gate.

Two consequences worth being explicit about:

1. More tests than scenarios. The inner loop typically writes many tests per scenario — one per concrete example, then regression tests as bugs surface. Acceptance criteria (§2.4) are the normative set the RFC is held to; the inner loop fills out the rest.
2. No refactor stage in the model. Refactoring is part of the inner loop, not a maturity stage. A Green or Validated RFC may be refactored without re-traversing the chain, as long as every §5 criterion stays green.

The split is the BDD/ATDD outer-shell convention adapted to a project already committed to Rust, proptest, and criterion: the scenarios are written in the BDD-flavoured prose of §2.1 because they live in RFCs and humans read them; the tests are written in the TDD-flavoured loop developers already know.

4. Entry points

The same machinery, three doors:

• Invariant entry — an item in CLAUDE.md §3. The criteria live in the RFC that operationalises that invariant; if no RFC yet exists for the relevant subsystem, the invariant is a known debt and the next RFC for that subsystem must address it.
• Hazard entry — an item in hazards.md. Each hazard’s Mitigation section names the RFCs and crates responsible; their acceptance criteria must reference the hazard id.
• RFC entry — a new RFC under docs/rfcs/. The RFC enumerates the invariants and hazards it touches in its §1 Summary; criteria in §5 must cover each.

5. Relationship to benchmarks.md

Correctness gates live here. Thesis-gates live in benchmarks.md §7. An RFC reaches Validated only when both:

• Every §5 acceptance criterion has a passing test, and
• Every thesis-gate in benchmarks.md §7 that the RFC’s pillars touch passes on representative corpora.

Single sentence; intentional non-duplication. benchmarks.md stays the performance owner.

6. Worked example

A concrete trace of the chain in §1, against an artefact that already exists. RFC 0001 Template miner is currently status: draft (becoming drafted once the amendment to docs/rfcs/README.md lands). Its operationalisation of CLAUDE.md §3.1 No silent template merges and hazards.md H1 Template miner correctness is the first place this process gets to bite on real material.

6.1 Invariant → RFC

CLAUDE.md §3.1 promises:

A template merge that crosses semantic boundaries (e.g. merging “user logged in” with “user logged out” because they share token structure) corrupts the backend.

hazards.md H1 names the canonical horror — user logged in <*> and user logged out <*> differing in one token, merging under a permissive threshold to user logged <*> <*>, a query for the login event silently returning logout rows.

RFC 0001 §6.4 Merge policy is the section that defends the invariant. As of the Drafted gate it commits to “When two templates become candidates for merge”, an audit event schema, and the rule “Default: strict. Never silent. No exceptions.”

6.2 RFC → Acceptance criteria

The Specified gate adds a new §5 to RFC 0001:

Scenario H1.1 — Semantically distinct templates do not silently merge

• Given a corpus containing user logged in <*> and user logged out <*>
• When similarity threshold is 0.7 (default)
• Then the two remain distinct template_ids
• And any widening produces an audit event recording both old and new templates

Scenario H1.2 — Lossy-zone match retains body

• Given a line whose best match has confidence in the lossy zone (floor ≤ x < threshold)
• When the line is ingested
• Then the body column contains the original line bytes
• And the row carries lossy_flag = false (the flag is reserved for tokenizer / preprocessing failure per docs/rfcs/0001-template-miner.md §6.6 — the lossy zone retains the body but reconstruction still succeeds)

Scenario H1.3 — Every widening emits an audit event

• Given any sequence of inputs that triggers a template widening
• When the widening completes
• Then an audit event exists naming the old template, the new template, the tenant id, the timestamp, and the event_type

Three scenarios cover §3.1’s three rules: do not merge across semantics, retain bodies on low confidence, audit every widening. Reviewers ratify that this is exhaustive against CLAUDE.md §3.1 and H1; they do not catalogue every edge-case test the implementation will write.

6.3 Acceptance criteria → Red tests

The Red gate adds three stubs to crates/ourios-miner/tests/:

#![allow(unused)]
fn main() {
/// Scenario H1.1 — Semantically distinct templates do not silently merge.
/// See `docs/rfcs/0001-template-miner.md` §5.
#[test]
#[ignore = "RFC 0001 Red gate — implementation pending"]
fn h1_1_login_and_logout_remain_distinct_at_default_threshold() {
    todo!("RFC 0001 §6.4");
}

/// Scenario H1.2 — Lossy-zone match retains body.
/// See `docs/rfcs/0001-template-miner.md` §5.
#[test]
#[ignore = "RFC 0001 Red gate — implementation pending"]
fn h1_2_lossy_zone_match_retains_body() {
    todo!("RFC 0001 §6.6");
}

/// Scenario H1.3 — Every widening emits an audit event.
/// See `docs/rfcs/0001-template-miner.md` §5.
#[test]
#[ignore = "RFC 0001 Red gate — implementation pending"]
fn h1_3_every_widening_emits_an_audit_event() {
    todo!("RFC 0001 §6.4");
}
}

Default cargo test skips the ignored stubs and passes (outer loop / CI green); cargo test -- --ignored exits non-zero with all three failing (inner loop / Red signal). The gate is satisfied; implementation may begin.

6.4 Red → Green

Implementation lands across ourios-miner (and supporting types in ourios-core). The three stubs become real tests: H1.1 ingests the two-template corpus, asserts two distinct template_ids, and queries the audit log for absence of widening events. H1.2 ingests a line whose token similarity falls in the lossy zone and asserts that the row’s body carries the original bytes and lossy_flag is false (the flag is reserved for the H7 reconstruction-failure case; see RFC 0001 §6.6). H1.3 ingests a sequence that provokes a widening and asserts the audit event’s structure.

cargo test --all-features passes. Reviewers confirm each H1.x id now resolves to a passing test via grep. No benchmark claim is made.

6.5 Green → Validated

benchmarks.md C2 Template count convergence is the thesis-gate that H1 most directly touches: if the miner is silently merging across semantics, template count grows wrong. The benchmark harness runs C2 on the LogPAI corpora and any self-collected corpus available, plots template count vs. lines ingested, and asserts the convergence target.

Once C2 passes — and any other thesis-gate the RFC’s pillars touch — the maintainer signs off. RFC 0001’s status: flips to accepted. The miner’s contract is now binding.

6.6 The failure mode that re-opens the RFC

A hypothetical: six months in, three independent PRs land that each add a workaround to keep H1.1 green — a special-case for common verb pairs, then for HTTP method tokens, then for log-level tokens. Each workaround is small, each test stays green. By the fourth PR, a reviewer notices: the criterion has stopped being a defence and has become a moving target. Per §3.1, the RFC reopens. The right answer is not a fifth workaround; it is to revisit RFC 0001 §6.4 — the merge policy itself is under-specified for the workloads we are seeing.

This is what the threshold in §3.1 is for. It is not a CI-flake counter; it is a signal that the design’s defence has eroded and needs to be redrawn before more code is written on top of it.

7. What this doc is not

• Not test-tooling guidance — proptest, criterion, etc. live in CLAUDE.md §6.2.
• Not a coverage policy — Ourios is a correctness project; line coverage is the wrong metric.
• Not an agent-instruction file — agents follow it because it is written down, not because it speaks to them.

8. Resolved decisions

Three questions raised during the outline review, decided before expansion so the rationale is preserved:

• Maturity stages appear in RFC frontmatter as the status: field. Reviewers and tooling see an RFC’s current stage without reading the body. See §3.
• Single regressions do not reopen a Validated RFC. A failing test on an existing criterion is the test doing its job; standard PR workflow applies. Repeated regression on the same criterion (rough threshold: same scenario id failing on three independent commits, or two distinct regressions touching the same criterion, both measured in a 30-day rolling window) signals the criterion has stopped being a defence and reopens the RFC. See §3.1.
• Thesis-gate failures during Validated follow benchmarks.md §7, not this doc. One thesis-gate failing on one corpus → tuning RFC; two or more → pillar RFC and an implementation pause.

Proposed amendment — docs/rfcs/README.md

Two changes. Shown as the new text:

In Required frontmatter

Update the status field’s valid values from the current four-state list to the five-stage maturity model plus terminals:

status: drafted | specified | red | green | validated | accepted | rejected | superseded

The maturity stages (drafted through validated) are gates an RFC moves through; accepted is the terminal post-maintainer-signoff binding state; rejected and superseded are the off-ramps. See docs/verification.md §3.

In Required sections

Insert a new item between the current §4 Alternatives considered and §5 Testing strategy, renumbering subsequent items:

5. Acceptance criteria — normative scenarios, one per invariant or hazard the RFC touches. Format: structured prose with Given / When / Then / And leading clauses; each scenario carries an id of the form H1.1, §3.4.2, or RFC<NNNN>.<m>, referenced from the test code so the mapping is greppable. See docs/verification.md §2.

Testing strategy shifts to §6, Open questions to §7, and References to §8.

In Lifecycle

Replace the current four-status list with the five-stage maturity model:

1. Drafted — PR opened with status drafted. Sections §§1–4 and §§7–8 are filled. Discussion happens in PR review.
2. Specified — §5 acceptance criteria are written, every invariant and hazard the RFC touches has at least one scenario, and review has confirmed the criteria are testable in principle.
3. Red — test stubs exist and fail. Implementation may begin.
4. Green — all acceptance criteria pass; unit + property + corpus tests green.
5. Validated — thesis-gates in docs/benchmarks.md §7 pass on representative corpora. Maintainer flips status to accepted.

A regression detected after Validated either reopens the RFC (if a criterion is invalidated) or spawns a tuning RFC per benchmarks.md §7 (if a thesis-gate degrades). See docs/verification.md §3.

The earlier superseded and rejected entries remain unchanged.

Existing RFC frontmatter

RFC 0001 and RFC 0002 currently carry status: draft. The amendment PR renames both to status: drafted so the maturity model applies uniformly. No content change to the RFCs themselves at that step.

Proposed amendment — CLAUDE.md

A single new subsection under §5 Development workflow, following §5.5 One-word mode:

5.6 Verification process

The path from invariant or hazard to passing test is described in docs/verification.md. Acceptance criteria live in RFC §5; docs/rfcs/README.md defines the maturity stages an RFC moves through. The shortest version of the rule: if a criterion cannot be turned into a test, the RFC has a gap.

No change to §6.2 Testing discipline; verification.md links to it. The §6.2 content (proptest, corpus tests, crash recovery, criterion) is the catalogue of techniques; verification.md is the process that decides which technique is required where.

Applying the amendments

The body of this document is the verification process spec. The two proposed amendments above are pending application:

• docs/rfcs/README.md — status: value list, new §5 Acceptance criteria in Required sections with renumbering, lifecycle rewrite, draft → drafted rename in RFC 0001 and 0002.
• CLAUDE.md — new §5.6 Verification process.

Both should land in a single PR. RFC 0001 then gets a §5 Acceptance criteria applied as the first concrete use of the process — the worked example in §6 of this document is the target shape, and applying it will probably surface specificity gaps in RFC 0001’s existing design. That surfacing is the point.

Add this document to docs/SUMMARY.md under the Architecture header in the same PR that applies the amendments.

Glossary

Vocabulary used in the Ourios docs. Entries marked (Ourios) carry a project-specific meaning that may differ from the industry-default. Cross-references in italics point to other entries here.

Audit event. A structured record emitted by the miner every time a template is widened (parameters generalised), merged with another template, or versioned. Audit events are themselves stored as logs and are queryable. They are the trail by which an operator can answer “did this template silently change yesterday?” See hazards.md H1.

Bit-identical reconstruction. The property that, for any ingested log line, either Ourios can reproduce the exact original byte sequence from what it stored, or the row carries lossy_flag = true. Never an in-between. Tested as a property test against the corpus. See CLAUDE.md §3.3, hazards.md H7.

Body. The free-form text content of a log record. In OTel terms, the body field of a LogRecord. In Ourios storage, the body is either reconstructible from template + params (most rows) or retained verbatim in a dedicated column (lossy rows, parse failures, or tenants who opted in to always-retain).

Compaction. Background process that merges many small Parquet files into fewer large ones, targeting row-group sizes of 128 MB to 1 GB and file sizes of 256 MB to 2 GB. Driven by the small-file hazard (H4).

Confidence. A scalar in [0, 1] assigned by the miner to each matched row, measuring how well the row matched its assigned template. The three-zone model partitions confidence into clean match (≥ threshold), lossy match (floor ≤ x < threshold), and parse failure (< floor). (Ourios) — extends Drain, which is binary-classifying.

Corpus. A collection of anonymised log lines used as test input. Lives under testdata/corpus/. Public LogPAI corpora form the floor; self-collected corpora per deployment archetype are added over time. Reconstruction, template-count convergence, and merge rate are all measured against the corpus on every miner change.

DataFusion. The Apache project providing the query engine Ourios uses. Ingests logical plans, optimises them, executes against Parquet. Ourios extends DataFusion with two custom logical nodes (render, template_id.resolves_to) but otherwise treats it as a black box. DataFusion specifics never leak into the user-facing DSL (H6).

Drain. The 2017 paper (He, Zhu, Zheng, Lyu — ICWS 2017) that introduces a fixed-depth tree algorithm for online log parsing. The basis of the miner. See docs/rfcs/0001-template-miner.md and docs/talks/0001-template-miner.md.

Drain3. The IBM-maintained fork of Drain that adds persistent state, masking, variable-length wildcards, and dynamic thresholds. Some of its extensions are adopted in Ourios; some are explicitly not. RFC 0001 §4 lists the per-extension verdict.

Drift. The phenomenon where a service’s log format changes between deploys, producing new templates that are aliases of older ones. (Ourios) — drift is detected as a first-class query, not an after-the-fact discovery. See H5 and RFC 0001 §6.7.

DSL. The user-facing query language for Ourios logs (RFC 0002). Compiles to DataFusion logical plans; does not expose SQL. Two candidate predicate sublanguages (OTTL-borrowed vs. distanced) and three top-level surfaces (SQL-clause, LogQL-pipe, Insights-verb) are under design.

Floor. The lower bound of confidence below which the miner declares a parse failure. Default ~0.3. Below the floor, the row is stored body-only and parse_failures_total increments. (Ourios) — not present in the Drain paper.

Fsync. The POSIX call that forces buffered writes to durable storage. The WAL fsyncs before acknowledging an OTLP batch. See H3, CLAUDE.md §3.4.

Hazard. A named failure mode that, if not actively mitigated, silently corrupts data or destroys the project’s value proposition. The seven current hazards are catalogued in docs/hazards.md. New hazards are added via a meta: RFC.

Ingester. The Ourios role that receives OTLP over gRPC/HTTP, mines templates, writes to the WAL, and (eventually) flushes to Parquet via the compactor. One half of the ingester/querier binary split.

Length group. The first-level partition in the Drain parse tree: one branch per distinct token count. Drain assumes lines of different length are probably from different call sites and uses length as a cheap initial filter.

Log group. Drain’s term for a template together with the rows that have matched it. A leaf in the parse tree contains a list of log groups.

Lossy. A row whose lossy_flag is set, indicating that reconstruction from template + params may not be byte-identical. Always paired with the original body being retained on that row. See H7.

LogPAI. The benchmark-corpus project for log parsing (github.com/logpai/logparser). Ourios uses LogPAI corpora (HDFS, BGL, Spark, Apache, OpenSSH, Windows) as the public-corpus floor for benchmarks.

Masking. Pre-tokenisation regex rules that replace volatile sub-strings (IPs, UUIDs, numbers) with placeholders before the miner walks the tree. A Drain3 extension. Whether and where Ourios applies masking is a design choice in RFC 0001 §4.

Merge. When the miner widens an existing template to absorb a new line — e.g. replacing a literal token with a wildcard. Every merge fires an audit event. Strict thresholds make merges rare; audit makes them visible. See H1.

Miner. Short for template miner — the Ourios subsystem that runs Drain online over ingested log lines and emits (template_id, params, confidence, lossy_flag) for each row. Lives in the ourios-miner crate. Designed in RFC 0001.

OTLP. OpenTelemetry Protocol, the wire format for telemetry data. The Ourios ingest contract: incoming logs are OTLP over gRPC or HTTP. We do not invent our own format.

OTTL. OpenTelemetry Transformation Language, the OTel Collector’s text-based DSL for filtering and mutating telemetry in processor pipelines. Ourios deliberates between borrowing OTTL’s predicate sublanguage and distancing from it (RFC 0002).

Parquet. The Apache columnar file format Ourios uses for on-disk storage. Per-column compression, predicate pushdown via min/max statistics, bloom filters, page indexes. The on-disk truth of the system; local disk is cache and WAL only. See CLAUDE.md §§2.1, 3.6.

Params. The variable parts of a log line that the miner extracts when matching a template. Bounded per-parameter to 256 B by default; overflow spills to the body column. See H2.

Parse failure. A row whose match confidence falls below the floor. Stored body-only; parse_failures_total counter increments.

Predicate pushdown. A query-engine optimisation where filter predicates are applied as early as possible — at the storage layer rather than after a full scan. Parquet’s min/max page statistics make this nearly free for time-range and equality filters. The mechanism by which predicate queries beat zstdcat | grep (B1).

Property test. A test that asserts an invariant over many randomly-generated inputs (typically via proptest). In Ourios: reconstruction is always a property test; the parser round-trips; the miner’s tree operations preserve invariants. See CLAUDE.md §6.2.

Querier. The Ourios role that accepts queries (over the DSL), plans them through DataFusion, scans Parquet, and returns results. Other half of the ingester/querier split.

Reconstruction. The act of producing the original body of a log line from the stored template + params (and, where retained, the captured whitespace state). Subject to the bit-identical guarantee. See H7.

Row group. Parquet’s unit of compression and predicate-pushdown locality — a horizontal partition of rows within a file. Target size 128 MB to 1 GB. Smaller row groups mean faster row-group skip but worse compression and more metadata overhead.

Similarity. The Drain match score between an incoming line and a log group’s template: the fraction of token positions where the line matches the template (wildcards count as matches). The single most important knob in the system. See RFC 0001 §3.

SUMMARY.md. mdBook’s table-of-contents file (docs/SUMMARY.md) that defines book navigation. Drafts (no link target) appear as greyed-out entries.

Template. The structural pattern of a class of log lines, with variable parts replaced by wildcards. E.g. ERROR db connection failed for user <*>. The miner extracts templates online from raw logs. (Ourios) — every template is scoped per tenant; the same string in two tenants is two templates.

Template id. The identifier of a template within a tenant. Either a hash of the canonical template string or a per-tenant monotonic integer (open question, RFC 0001 §6.1).

Template tree. The Drain parse tree, scoped per tenant. Its shape is root → length group → token-prefix nodes (depth d) → leaf log groups. (Ourios) — Drain assumes one tree; we keep one per tenant (CLAUDE.md §3.7).

Template version. A monotonic integer that bumps when a template’s representation changes (e.g. token order, new wildcard). The logical identity of the template persists across versions via the alias mechanism. See drift, RFC 0001 §6.7.

Tenant. An isolation boundary: a customer, a project, an environment. Every code path that touches data takes a tenant_id; every Parquet file is partitioned by tenant; every template tree is scoped per tenant. Multi-tenancy is not bolted on (CLAUDE.md §3.7).

Thesis-gate. A benchmark goal whose failure on representative corpora invalidates an architectural pillar — meaning the response is an RFC to revisit CLAUDE.md §2, not a tuning sprint. The five thesis-gates are catalogued in docs/benchmarks.md §7.

Threshold (st). The Drain similarity cutoff above which a line is assigned to an existing log group rather than opening a new one. Ourios default ≥ 0.7; values below 0.7 require an RFC (H1, CLAUDE.md §3.1).

Token-prefix node. Drain’s intermediate tree level: branches on the value of the line’s first N tokens (depth d, paper default 3–4). Below it, at the leaf, is a list of log groups.

Truncation marker. The placeholder that replaces an oversized params slot when the per-parameter byte limit is exceeded. The original value spills to the body column. See H2.

WAL. Write-ahead log. The Ourios ingester writes every acknowledged batch to the WAL, fsyncs, and only then acknowledges to the OTLP client. WAL segments are eventually flushed to Parquet by the compactor. The crash-recovery test SIGKILLs the ingester mid-batch and asserts no acknowledged data is lost. See H3, CLAUDE.md §3.4.

Benchmarks

Referenced from CLAUDE.md §6.2 (“regressions block merges”) and from docs/rfcs/0001-template-miner.md §8. Flat-file, living document, parallel to docs/hazards.md. Updated with measured results as they come in.

This document is an honesty contract with ourselves. The thesis (CLAUDE.md §2) claims that Parquet + Drain-derived template mining + DataFusion beats the naive alternative of byte-level compression over flat text. That claim is falsifiable. This file lists the measurements that would falsify it.

The thresholds were pinned before any number was measured; if we miss them on representative corpora, the thesis is wrong and a pillar changes. As of 2026-06-14 the four gating thesis-gates B1, B2, C1, C2 all pass on the §1 hardware baseline (§9.4/§9.6). A1 fails but no longer gates — RFC 0011 (accepted) reclassified the compression-vs-zstd ratio as a recorded diagnostic (its failure is structural; see §2 / the §7 table).

0. How to read this document

Every goal below carries two labels.

• Scope — thesis-gate, tuning-goal, or diagnostic.
  • A thesis-gate failing on representative corpora means a pillar (CLAUDE.md §2) is wrong. The response is an RFC, not a sprint.
  • A tuning-goal failing means the design is sound but the implementation needs work. The response is a PR.
  • A diagnostic is measured and recorded but gates nothing — it characterises a property or guards against regression. A1 was reclassified here by RFC 0011 (accepted); see §2.
• Bar — must-win, should-win, stretch, or informational.
  • must-win — shipping without it is shipping a broken claim.
  • should-win — expected on representative corpora; explained when missed.
  • stretch — aspirational; missing is not a bug.
  • informational — a diagnostic’s bar: the number is recorded for insight, never blocks.

A goal with scope thesis-gate and bar must-win is load-bearing for the whole project. Four of those below are gating — B1, B2, C1, C2, each marked [THESIS]. A1 keeps the [THESIS] tag (a thesis-relevant measurement) but RFC 0011 (accepted) set its scope to diagnostic: it is recorded, not gating (see its section below and the §7 table).

1. Corpora and methodology

Before any goal is meaningful, the corpora and methodology must be pinned — otherwise we will argue about numbers instead of about architecture.

• Public: LogPAI corpora (HDFS, BGL, Spark, Apache, OpenSSH, Windows) — the same corpora the Drain paper reports on. Lets us reproduce published claims as a sanity floor.
  • LogHub HDFS_v1 is the first of these wired in, as a bench-time-fetched corpus for the query gates: .github/workflows/query-bench.yml downloads HDFS_v1.zip from the official Zenodo record (record 8196385, DOI 10.5281/zenodo.8196385, md5-pinned in the workflow), uses the extracted HDFS.log (~1.47 GiB plain text — above §8’s ≥ 1 GiB canonical minimum) in-job, and discards it with the runner. It is never redistributed: not committed (the testdata/corpus/README.md anonymisation gate — LogHub data is explicitly not sanitised), not attached to a release, not uploaded as an artifact; only aggregate numbers leave the job. LogHub’s license notice, included here as it requires: “The datasets are freely available for research or academic work. For any usage or distribution of the datasets, please refer to the loghub repository URL (https://github.com/logpai/loghub) and cite the loghub paper: Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, Michael R. Lyu. Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics. In IEEE International Symposium on Software Reliability Engineering (ISSRE), 2023. The above license notice shall be included in all copies.”
• Self-collected (deferred): at least one anonymised corpus per target deployment archetype. Proposed set:
  1. Structured Java/Spring service (well-templated, low entropy).
  2. Go service under Kubernetes (heterogeneous, mid entropy).
  3. Heterogeneous k8s aggregate across many services (high entropy, mixed formats).
• Hardware baseline: a commodity 8 vCPU / 32 GiB RAM host with gp3-class SSD. All must-win numbers are quoted against this baseline; scaling to larger hardware is a separate question. The realised baseline (the baseline-8vcpu-32gib hardware tag, first used for the §9.4 authoritative run) is a dedicated host with 8 dedicated vCPU, 32 GiB RAM, and a local NVMe-class SSD — at or above the spec on every axis, so numbers quoted against the tag satisfy this baseline. It is identified only by the tag.
• Reference system: zstdcat <file.zst> | grep <pattern>. The “naive alternative” the thesis beats or does not beat. Everything is quoted relative to this, not in absolute terms.

Goals quoted below assume this setup. When a goal is measured on a different setup, the measurement is annotated.

2. Compression goals (Category A)

The core claim that template mining does useful work before byte codecs run.

A1 [THESIS] — End-to-end compression ratio vs. zstd-alone

Demoted to a diagnostic (RFC 0011, accepted). A1 is refuted on every corpus class — including the maximally-templated one — for structural reasons, so it no longer gates any RFC’s validated. It is still measured and recorded (§7 table / §9 series) as the columnar queryability premium and a codec-regression guard. The scope, bar, target, and falsifier below are retained as the diagnostic’s reference line — now informational, not gating.

• Scope: diagnostic (RFC 0011; originally thesis-gate).
• Bar: informational (RFC 0011; originally must-win).
• Metric: bytes(raw_corpus) / bytes(ourios_parquet_directory) compared to bytes(raw_corpus) / bytes(zstd_compressed_corpus).
• Target: Ourios ratio ≥ 3× the zstd-alone ratio, on every corpus in §1. Best-case corpora (well-templated services) should show ≥ 10×.
• Falsifier: if any representative corpus yields ≤ 2× improvement over zstd-alone, the template-mining pillar is not pulling its weight on that class of logs. Open an RFC.
• Why recorded (diagnostic, not a bar): CLAUDE.md §2 pillar #2 describes a logical 50–200× reduction (lines → (template_id, params)) whose payoff is query pruning (B1/B2), not on-disk bytes vs a byte codec. A1 tracks the on-disk ratio as the columnar queryability premium + a codec-regression guard; RFC 0011 (accepted) demoted it from a gate to this diagnostic.

A2 — Bytes per line, amortised

• Scope: tuning-goal.
• Bar: should-win.
• Metric: total Parquet bytes for tenant / line count for tenant.
• Target:
  • Structured service logs: ≤ 30 B/line.
  • Heterogeneous k8s: ≤ 100 B/line.
  • Stretch: ≤ 15 B/line on high-repetition corpora.
• Why: makes A1 legible to operators, who think in bytes-per-line, not ratios.

3. Query performance goals (Category B)

Why not zstdcat | grep? Because the query layer is supposed to exploit structure the tree extracted.

B1 [THESIS] — Predicate-pushdown queries

• Scope: thesis-gate.
• Bar: must-win.
• Query shape: count events WHERE tenant=X AND ts BETWEEN t1 AND t2 AND level='ERROR'.
• Reference: zstdcat files_in_range.zst | grep ERROR | wc -l on the same corpus, same time window.
• Target: Ourios ≥ 10× faster at 1 GiB corpus, widening to ≥ 100× at 100 GiB.
• Falsifier: if Ourios is not materially faster than the zstdcat pipeline on predicate queries, DataFusion + Parquet statistics are not delivering on the “skip row groups via footer reads” pillar (CLAUDE.md §2.1). Open an RFC.
• Instruments: B1 is proven structurally (deterministically) by ourios-querier’s rfc0007_1_* tests. The criterion bench crates/ourios-bench/benches/b1.rs adds the wall-clock ratio: a b1/synthetic group (controlled pruning instrument vs. an in-process zstdcat | grep reference) and a b1/real-corpus group (set OURIOS_B1_CORPUS_DIRS to a comma-separated list of corpus dirs; skipped when unset). The real arm runs OTLP corpora only (corpus/otel-demo-v*, which carry real per-record severity): B1’s predicate filters on severity, and the RFC 0006 §3.3 plain-text loader assigns every line a fixed severity (9 / INFO), so a severity predicate over a plain-text corpus has no selectivity and such dirs are skipped with a note. CI runs land via .github/workflows/query-bench.yml on ci-runner — indicative only. The authoritative numbers are the baseline-8vcpu-32gib run of 2026-06-12 (§9.4): PASS at 34.2× / 25.4× on the two ~1 GB OTel-Demo corpora, with exact row-count agreement against the reference pipeline. Open quality improvement (non-blocking): the measured error bands are ultra-thin (11 / 28 rows), which flatters pruning — a denser error band is the remaining methodological wish.

B2 [THESIS] — Template-exact queries

• Scope: thesis-gate.
• Bar: must-win.
• Query shape: SELECT * WHERE template_id = X AND ts BETWEEN ….
• Target: latency proportional to result cardinality, not to corpus size, above a corpus size of ~10 GiB. Concretely: median latency ≤ 200 ms for a query returning 10 000 rows, regardless of whether the corpus is 10 GiB or 10 TiB.
• Falsifier: if template-exact queries scan proportionally to corpus size, template mining is buying compression but not query locality — the inverted-index collapse thesis (CLAUDE.md §2) is wrong in practice. Open an RFC.
• Instruments: B2 is proven structurally (deterministically) by ourios-querier’s rfc0007_2_* test — for a fixed result the scanned row groups + bytes stay flat as the corpus grows. The criterion bench crates/ourios-bench/benches/b2.rs adds the wall-clock view: a b2/synthetic group (result held constant, corpus scaled 1×/10×/50×) and a b2/real-corpus group over real corpora (set OURIOS_B2_CORPUS_DIRS to a comma-separated list of corpus dirs; skipped when unset, since the corpora aren’t committed). Both loader formats feed it: the OTLP/JSON corpus/otel-demo-v* releases and the bench-time-fetched plain- text LogHub HDFS_v1 (§1). Run with cargo bench -p ourios-bench --bench b2. CI runs land via .github/workflows/query-bench.yml on ci-runner — indicative only. The authoritative numbers are the baseline-8vcpu-32gib run of 2026-06-12 (§9.4): PASS — the windowed template-exact scan stays at 1 row group with a flat ~4.2–5.9 ms latency band across every corpus, including the first reading from a second corpus family (LogHub HDFS_v1, 11.2 M rows: 1/14 row groups, 5.92 ms), while the full-span variant grows with corpus size. The formal target speaks above ~10 GiB, which remains a future scale extension; the flat shape holding at 11.2 M rows across two corpus families is the operative evidence.

B3 — Substring queries (the hard case)

• Scope: tuning-goal.
• Bar: must-match; stretch: beat.
• Query shape: SELECT * WHERE body LIKE '%<substring>%' or equivalent.
• Target: not slower than the reference system. Stretch: faster on well-templated corpora by searching the template text rather than every line.
• Why this is only tuning-goal, not thesis-gate: substring search is the case where the tree does not help directly. We are allowed to match the reference system here; losing against it is a bug but not a pillar failure.

4. Miner correctness goals (Category C)

Correctness is not a performance goal, but it belongs here because these are the properties the benchmark harness actually measures on every run.

C1 [THESIS] — Bit-identical reconstruction rate

• Scope: thesis-gate.
• Bar: must-win.
• Metric: of all non-lossy-flagged rows, fraction whose reconstruct(template, params) equals the ingested bytes exactly.
• Target: 100.000%.
• Falsifier: a single row that reconstructs wrong without a lossy flag is a violation of CLAUDE.md §3.3 and a blocker, not a benchmark regression. Accompanied by: the lossy-flagged fraction should be ≤ 5% on structured corpora, ≤ 20% on heterogeneous ones, as a quality signal (not a gate).
• Why this is a thesis-gate: if we cannot promise reconstruction, the honesty contract (lecture §6) collapses.

C2 [THESIS] — Template count convergence

• Scope: thesis-gate.
• Bar: must-win.
• Metric: template count as a function of lines ingested, on a corpus from a single stable service.
• Grain (amended for #444, 2026-07-10): because the metric is defined per stable service, the gate is evaluated per service.name on a multi-service corpus, not on the whole corpus. A corpus passes iff every service with ≥ 1 M lines converges; a single-service (or plain-text <unknown>) corpus is gated on that one service’s exact-millionth-line ratio, reproducing the pre-amendment verdict for historical converged corpora. The whole-corpus ratio is retained as a diagnostic. See RFC 0006 §3.4.3.
• Target: template count grows sub-linearly and plateaus within 2× of its steady-state value by 1 M lines. Steady-state value is corpus-specific but is on the order of 10²–10⁴ templates for a normal service.
• Falsifier: if template count grows linearly with corpus size, Drain has failed to abstract — we are storing one template per line, which means the tree is providing compression only accidentally. That is the inverse of the thesis. Open an RFC.

C3 — Merge rate

• Scope: tuning-goal.
• Bar: should-win.
• Metric: merges_total / lines_ingested.
• Target: ≤ 1 merge per 10⁵ lines on stable corpora, with every merge carrying an audit event. Spikes above this rate are investigated; they usually indicate a new service version.
• Why only tuning-goal: merge rate depends on corpus stability more than on algorithm quality. The auditing is the invariant (§3.1); the rate is a signal.

C4 — Parameter overflow rate

• Scope: tuning-goal.
• Bar: must-win.
• Metric: fraction of rows where any params slot hit the 256 B limit.
• Target: ≤ 1% on representative corpora, per CLAUDE.md §3.2.
• Falsifier (tuning sense): if >1% on a common archetype, either the limit is too tight for that workload or a masking rule is missing. The response is tuning, not an RFC.

5. Ingest goals (Category D)

The hot path must keep up with real deployments; otherwise none of the above matters.

D1 — OTLP → WAL throughput

• Scope: tuning-goal.
• Bar: must-win.
• Metric: lines/second/core sustained, with WAL fsync batched at 100 ms (the CLAUDE.md §3.4 default).
• Target: ≥ 100 000 lines/s/core, with p99 ingest-ack latency ≤ 200 ms.
• Falsifier (tuning sense): below this we cannot ingest a meaningful share of production traffic per node, which makes the operational story uninteresting.

D2 — WAL → Parquet compaction keeps up

• Scope: tuning-goal.
• Bar: must-win.
• Metric: WAL backlog (bytes, segments) as a function of time under sustained ingest at D1’s rate.
• Target: bounded; backlog returns to zero during any one-hour window of sustained load.
• Falsifier (tuning sense): a growing backlog under steady-state load means compaction is the bottleneck — a correctness-adjacent bug because it lets the WAL grow unboundedly.

D3 — Small-file count under sustained load

• Scope: tuning-goal.
• Bar: should-win.
• Metric: number of Parquet files per tenant per day after background compaction has settled.
• Target: file sizes cluster in the 256 MiB–2 GiB band per CLAUDE.md §4 / hazard 4. Fewer than 5% of files below 128 MiB at steady state.
• Why: the small-file problem is a named hazard, not a nice-to-have.

6. Honesty goals (Category E)

Not performance. Not falsifiable by a benchmark in the usual sense. Listed here because the benchmark harness asserts them on every run.

E1 — Zero silent merges

• Scope: correctness invariant (not a benchmark).
• Metric: in the corpus-test suite, for every row whose template_id changed over its lifetime in the tree, an audit event exists with matching timestamp and tenant.
• Target: 100%. This is a proptest, not a measurement.

E2 — Zero cross-tenant leakage

• Scope: correctness invariant (not a benchmark).
• Metric: no template mined under tenant A ever appears in tenant B’s tree or in a row for tenant B.
• Target: 100%. Asserted via corpus tests that interleave lines from two synthetic tenants and verify complete isolation.

7. The thesis-gate summary

The five [THESIS]-tagged goals, consolidated:

Policy: if one thesis-gate fails on one representative corpus, that is a corpus-specific tuning RFC. If two or more thesis-gates fail on any representative corpus, that is a pillar-level RFC — we pause implementation and revisit CLAUDE.md §2 before continuing.

This escalation rule is the point of the whole document. The worst failure mode for a greenfield project is shipping something whose central claim quietly fails on real data and then papering over it with more implementation. These goals exist so we cannot do that to ourselves without noticing.

8. What is deliberately out of scope

• SIEM-style full-text search latency — explicitly out of scope (CLAUDE.md §1).
• Cross-tenant aggregation queries — tenancy is isolation-first (CLAUDE.md §3.7). Aggregations that cross tenants are an RFC topic, not a benchmark.
• LLM-based parser comparisons — interesting, deferred. Listed in RFC 0001 §7 as an alternative. Benchmarking it would be a separate RFC.
• Cold-start query latency — below a corpus size of ~1 GiB the overhead of Parquet metadata dominates, and the thesis is uninteresting. Benchmarks start at 1 GiB.

9. Status

First measurements landed 2026-06-01 (the writer-side gates A1 / C1 / C2 — see §9.1). They are diagnostic, not canonical: they ran on a GitHub-hosted runner (ci-runner), not the §1 hardware baseline (baseline-8vcpu-32gib), against an OTel-Demo corpus that is shape-representative (real multi-service template + envelope diversity) but not size-representative — every corpus is well below §8’s ≥ 1 GiB canonical minimum, so this run is intentionally diagnostic, not a thesis verdict. The query-side gates now have instruments — B1 and B2 are proven structurally in ourios-querier, and both have criterion latency benches with real-corpus arms (§B1/§B2 “Instruments”; OTel-Demo for B1, OTel-Demo + the bench-time-fetched LogHub HDFS_v1 for B2, run on ci-runner via .github/workflows/query-bench.yml as indicative numbers). 2026-06-11 extended the writer-side scale series to ~1 GB (§9.2) and landed the first B1/B2 query readings (§9.3) — recorded here as indicative ci-runner entries per the maintainer’s 2026-06-12 authorization. 2026-06-12 landed the authoritative baseline run (§9.4): every gate measured on the §1 hardware (baseline-8vcpu-32gib), recorded per the maintainer’s 2026-06-12 authorization. B1, B2, C1, and C2 pass authoritatively; on that basis RFC 0007 flipped to validated (its gates, per docs/verification.md §3, are the querier-pillar ones — B1/B2). A1 fails authoritatively and carries a hardware-sensitivity caveat (§9.4). (A1 was subsequently reclassified a recorded diagnostic, not a gate — RFC 0011, accepted 2026-06-14. The A1 readings throughout §9 are diagnostic; A1 gates nothing, and the “open gate” / “must-win” framing in the dated entries below is superseded.)

Reviewers: a PR that materially affects the hot path must either (a) cite the benchmark result and its delta against the relevant goal, or (b) explain why the hot-path effect is bounded below measurability. “I did not run the benchmarks” is a PR rejection, per CLAUDE.md §6.6.

No ourios-bench --update-benchmarks-md run has populated this region yet. It is the bench-managed results area — automated runs replace everything between these markers with one table per (git-sha, hardware). The hand-written §9.1 below is the curated diagnostic narrative and lives outside the region so automated runs never touch it. (This empty region is pre-placed so the first --update-benchmarks-md run replaces it in place rather than appending a second results section at end-of-file.)

9.1 Results — 2026-06-01 (diagnostic, ci-runner)

Corpus. corpus/otel-demo-v{1..4} — OTel Demo 2.2.0 logs captured via the collector fileexporter (workflow .github/workflows/capture-otel-demo-corpus.yml), business-service logs only (collector self-telemetry + load-generator filtered out), OTLP/JSON. Sizes 30 / 136 / 272 / 547 MiB — all below §8’s ≥ 1 GiB canonical benchmark minimum (this run is deliberately sub-minimum, to chart the trend, hence diagnostic). Hardware. ci-runner (hosted, ~4 vCPU) — not the §1 baseline, so deltas are indicative, not authoritative.

A1 — compression (target: ourios ≥ 3.0× zstd-19).

Scale series (ourios at the production ZSTD-3 default):

Codec sweep (v4 = 547 MiB, ourios ZSTD level varied):

A1 verdict: FAIL (target 3.0×; best observed 0.829). Both levers are bounded. Scale lifts the delta but plateaus ~0.78 (ourios asymptotes ~25×; zstd-19 is flat ~32× — the logs are locally repetitive, so zstd compresses them well at any size, not via a whole-corpus window). Raising ourios’s codec to ZSTD-19 adds only ~+0.07 and saturates by level 9. Even at equal codec strength, ourios stays ~17% larger than monolithic zstd-19: a structural cost of columnar Parquet (per-column/per-chunk framing, page indexes, row-group metadata, bloom filters) versus zstd-19 over one concatenated stream. That same chunking is what enables row-group skipping — so the ~17% space premium is the price of queryability, not an optimisation target. On pure compression of this corpus, ourios ≈ 0.83× zstd-19; the thesis rests on query performance (B1/B2), not on beating a byte codec.

C1 — reconstruction (target: 100% bit-identical or flagged lossy). PASS at every size: 1.0 reconstruct rate, ~1.1% of records flagged lossy (structured/kvlist bodies) and retained verbatim per CLAUDE.md §3.3.

C2 — template-count convergence (target: sub-linear). PASS (supportive). Templates grew 282 → 429 → 722 → 1322 while records grew 38k → 183k → 366k → 735k — sub-linear throughout. The formal gate abstains below 1 M lines (§3.4.3), but the curve shape is the strongest evidence yet for the template-mining premise.

Escalation (§7). One gate (A1) fails, on a size-non-representative corpus (all < §8’s 1 GiB minimum) and non-baseline hardware — so this is “corpus-specific,” not the two-gate pillar-level pause. C1 + C2 support the thesis. The production ZSTD-3 default is retained: the codec gain is small, saturates by level 9, and the residual gap is structural, so a higher default isn’t worth the ingest-CPU.

9.2 Results — 2026-06-11 (diagnostic, ci-runner) — A1 / C1 / C2 at ~1 GB

Corpus. corpus/otel-demo-v5 (1,042,274,219 B) and corpus/otel-demo-v6 (1,034,615,505 B) — same capture pipeline as §9.1, extending the scale series to ~1 GB (both within 4% of, but still just under, §8’s ≥ 1 GiB binary minimum). v6 was captured with the OTel Demo failure flags enabled (adFailure cartFailure productCatalogFailure), so it carries a real error band; v5 is an unflagged capture. Hardware. ci-runner — indicative, not the §1 baseline. Runs. bench.yml 27370641352 (v5), 27373716667 (v6).

A1 — compression (target: ourios ≥ 3.0× zstd-19).

A1 verdict: FAIL (target 3.0×). The scale series now reads 0.465 (v1, 30 MiB) → 0.666 (v2) → 0.725 (v3) → 0.758 (v4) → 0.828 (v5) / 0.824 (v6): the delta is size-driven and still rising, but decelerating — the crossover is not reached at ~1 GB, consistent with §9.1’s structural reading (ourios asymptotes ~26×; zstd-19 stays flat ~32×). v5 ≈ v6 shows the failure-flag error band does not perturb A1. This is the first A1 miss at (essentially) canonical size, so §9.1’s “size-non-representative” mitigation no longer applies; it remains a single-gate fail (no §7 two-gate pause), the §9.1 structural explanation stands, and the thesis-deciding counterpart — B1/B2 — now passes indicatively (§9.3). Whether the §7 corpus-specific tuning-RFC response triggers is a maintainer decision, sensibly taken once an authoritative baseline-8vcpu-32gib run confirms the number. (Resolved 2026-06-12: the §9.4 baseline run confirms — and slightly worsens — the deltas; the decision is now live with the maintainer.)

C1 — reconstruction (target: 100% bit-identical or flagged lossy). PASS on both: 1.000000 — v5 reconstructs 1,213,004 / 1,213,004 non-lossy rows exactly (lossy ratio 0.0114); v6 1,208,323 / 1,208,323 (lossy 0.0112).

C2 — template-count convergence (target: ratio ≥ 0.5 at 1 M lines). PASS on both — and for the first time on ≥ 1 M-line corpora, so the formal gate applies rather than §9.1’s abstention: v5 convergence ratio 0.756 (end count 1605, sample cadence 1336); v6 ratio 0.760 (end count 1606, cadence 1329).

9.3 Results — 2026-06-11 (indicative, ci-runner) — first B1 / B2 query readings

Corpus. corpus/otel-demo-v{4,5,6} (the §9.1 / §9.2 captures). The LogHub HDFS_v1 B2 arm did not run (fetch_hdfs off — memory-bound on the hosted runner), so only one corpus family has fed the query gates. Hardware. ci-runner — indicative, not the §1 baseline. Runs. query-bench.yml 27379085890 (B1 + the B2 structural metrics, after the effective-timestamp stack #178/#179) and 27357104694 (the prior run; its windowed / full-span latencies are quoted where noted). Recording. B1/B2 entries land in §9 per the maintainer’s 2026-06-12 authorization. RFC 0006 never reserved §9 (its §1 anticipated B1/B2 landing “in a follow-up extension PR once the querier is live” — RFC 0007); the workflow itself never writes §9 — every entry here is curated by hand.

B1 — predicate pushdown vs zstdcat | grep (target: ≥ 10× at 1 GiB). Query: severity ERROR, full corpus span. Run 27379085890:

Row counts agree exactly with the reference pipeline on both corpora. v4 is skipped: the unflagged 100-user capture genuinely contains zero error-band rows, so the predicate selects nothing.

B1 verdict: PASS (indicative) — both corpora clear the ≥ 10× bar at 3–4× margin, on the first real-corpus reading. Caveats, stated plainly: ci-runner, not the §1 baseline; the error bands are ultra-thin (11 / 28 rows — extreme selectivity is the friendliest case for pruning); both corpora sit just under the §8 1 GiB minimum. An authoritative baseline-8vcpu-32gib rerun (ideally with a denser error band) is required before this counts as the canonical B1 number.

B2 — template-exact latency ∝ result, not corpus. Windowed 1-hour template-exact query, result roughly constant as the corpus grows. Structural metrics (run 27379085890): scanned row groups stay flat at 1 — v4 1/5, v5 1/6 (17,632 rows, 1.86 MB), v6 1/6 (11,750 rows, 1.59 MB). Wall-clock (prior run 27357104694): windowed latencies sit in a flat ~3.4–4.1 ms band (v4 3.59 / v5 4.13 / v6 3.40 ms) while the full-span variant grows with corpus size (7.3 / 10.6 / 10.6 ms) — exactly the result-bound-vs-corpus-bound split the gate asks for.

B2 verdict: PASS (supportive, indicative) — the flat shape is confirmed on real corpora at ~1 GB; the formal target speaks above ~10 GiB, which remains unmeasured, and the second corpus family (HDFS_v1) hasn’t fed the arm yet.

RFC 0007 validated assessment. These are the measurements the RFC 0007 green → validated gate needs, but not yet in the form the ladder requires (§1 quotes must-win numbers against baseline-8vcpu-32gib): see the status note in docs/rfcs/0007-querier.md. The RFC stays green with a validated-pending note — authoritative baseline rerun required; denser error band and a second corpus family supporting. (Resolved 2026-06-12: the §9.4 authoritative run delivered the baseline rerun and the second corpus family (HDFS_v1); RFC 0007 is validated. The denser error band remains an open quality improvement.)

9.4 Results — 2026-06-12 (authoritative, baseline-8vcpu-32gib)

Corpus. corpus/otel-demo-v{1..6} (the §9.1 / §9.2 captures; 30 MiB → ~1 GB) for A1 / C1 / C2 and B1/B2’s OTel-Demo arms, plus — for the first time — the bench-time-fetched LogHub HDFS_v1 (§1; ~1.47 GiB plain text, 11,175,629 rows ingested across 5 files) feeding the B2 arm as the second corpus family. Hardware. baseline-8vcpu-32gib — the §1 baseline (8 dedicated vCPU, 32 GiB RAM, local NVMe-class SSD). These are the authoritative numbers the §1 methodology quotes must-win gates against; the §9.1–§9.3 ci-runner entries remain indicative history. Runs. Dedicated baseline host (no CI run id): one ourios-bench run per corpus (A1/C1/C2) plus one query-bench run (B1 + B2), executed 2026-06-11/12; raw logs retained by the maintainer. Recorded per the maintainer’s 2026-06-12 authorization.

A1 — compression (target: ourios ≥ 3.0× zstd-19).

A1 verdict: FAIL (authoritative) (target 3.0×; best observed 0.751). The delta is monotonic with corpus size and the crossover is unobserved, consistent with §9.1’s structural reading. One finding must be recorded honestly: the authoritative deltas sit below the ci-runner series (0.465 → 0.828) at every size — the ourios side compressed less effectively on this hardware (e.g. v5: 23.8× vs CI’s 26.3×) while zstd-19 stayed essentially stable (31.7× on both) — i.e. the ourios writer’s output is environment-sensitive (suspected row-group sizing / threading effects on the resulting encodings). That is now an open A1 investigation item alongside the structural gap itself. A1 gates the compression pillar (RFC 0006’s remit); the §7 escalation response is with the maintainer.

C1 — reconstruction (target: 100% bit-identical or flagged lossy). PASS (authoritative) on every corpus: 1.000000 throughout — v5 reconstructs 1,213,004 / 1,213,004 non-lossy rows exactly (lossy ratio 0.0114), v6 1,208,323 / 1,208,323 (lossy 0.0112); v1–v4 likewise 1.000000 (lossy 0.0097–0.0112). The formal ≥ 1 M-line gate passes on the baseline.

C2 — template-count convergence (target: ratio ≥ 0.5 at 1 M lines). PASS (authoritative) on both ≥ 1 M-line corpora: v5 ratio 0.756 (end template count 1605, sample cadence 1336), v6 ratio 0.760 (end count 1606, cadence 1329). v1–v4 abstain (< 1 M lines), as in §9.1.

B1 — predicate pushdown vs zstdcat | grep (target: ≥ 10× at 1 GiB). Query: severity ERROR, full corpus span. v4 is skipped (zero error-band rows, as in §9.3).

Row counts agree exactly with the reference pipeline on both corpora (11 and 28).

B1 verdict: PASS (authoritative) — both corpora clear the ≥ 10× bar at 2.5–3.4× margin on the §1 baseline. Remaining caveat, non-blocking: the error bands are still ultra-thin (11 / 28 rows — the friendliest case for pruning); a denser error band stays an open quality improvement.

B2 — template-exact latency ∝ result, not corpus.

Full-span template-exact (result grows with the corpus, so latency may too):

Windowed 1-hour template-exact (the gate’s shape: result roughly constant as the corpus grows):

The HDFS_v1 row is the first reading from the second corpus family (plain-text, the template-diversity case): the corpus is 8–15× the OTel-Demo row counts, yet the windowed scan still touches 1 row group (13 pruned) and stays inside the same flat latency band, while the full-span variant grows with the corpus (6.84 → 30.19 ms) — exactly the result-bound-vs-corpus-bound split the gate asks for.

B2 verdict: PASS (authoritative) — windowed ~10–28 k-row results answer in 4.2–5.9 ms (gate: ≤ 200 ms for ~10 k rows), flat from 735 k to 11.2 M rows across two corpus families. The formal target’s ≥ 10 GiB regime remains a future scale extension; the measured shape is the operative evidence.

RFC 0007 green → validated (resolved). The docs/verification.md §3 ladder reads: “Every thesis-gate in benchmarks.md §7 that the RFC’s pillars touch passes on representative corpora.” RFC 0007’s pillar is the query engine (pillar #3); its gates are B1 and B2, both now passing authoritatively on the §1 baseline over ~1 GB+ corpora including a second family. A1 does not gate RFC 0007 — it belongs to the template-mining/compression pillar, measured under RFC 0006. RFC 0007 is therefore flipped to validated (see its status note); accepted awaits maintainer sign-off per the ladder.

9.5 Results — 2026-06-13 (diagnostic, local unknown hardware) — A1 / C1 / C2 on HDFS_v1

Corpus. LogHub HDFS_v1 (Zenodo record 8196385, md5 76a24b4d…) — 11,175,629 lines, 1,577,982,906 raw bytes; fetched at bench time, never redistributed (query-bench.yml). The maximally-templated log corpus (a handful of templates over 11.2 M lines) — the single best case for the template-mining compression premise. Run via ourios-bench --gates a1,c1,c2 --parquet-zstd-level 19 --allow-unknown-hardware. Local hardware → diagnostic, not authoritative; A1’s verdict is corpus-structural and hardware-independent (compressed bytes are deterministic), C1/C2 are ratios, so the findings hold regardless of the runner.

A1 — the decisive finding (→ RFC 0011). A1 had only ever been measured on OTel-Demo (best 0.829×, §9.1/§9.4). HDFS_v1 is the corpus that should most reward template mining, yet A1 fails harder (0.516×): the more templated the corpus, the more completely monolithic zstd-19 captures its redundancy in one window (16×), while template mining’s extracted params (block IDs, timestamps, IPs) are high-cardinality columns that don’t compress as well and the columnar layout adds framing. The best case for template mining is the best case for the byte codec. So ≥ 3× over zstd cannot hold on any realistic log corpus — A1 is demoted to diagnostic and template mining’s compression value is recognised as logical/query-pruning (B1/B2), not on-disk bytes. See RFC 0011.

C1 + C2 — the miner pillar’s real gates, PASS on a representative corpus. At 11.2 M lines C1 is bit-identical (1.0) with a 4.6e-06 lossy ratio, and C2 plateaus at 40 templates with the formal gate applying (not abstaining, unlike the §9.1 sub-1 M runs). Under RFC 0011 these are RFC 0001’s validated thesis gates — both pass here. The authoritative baseline-8vcpu-32gib representative rerun (for the actual RFC 0001 validated flip) followed on 2026-06-14 (§9.6); as expected of deterministic verdicts, the numbers are identical.

9.6 Results — 2026-06-14 (authoritative, baseline-8vcpu-32gib) — C1 / C2 on HDFS_v1

Corpus. LogHub HDFS_v1 (Zenodo record 8196385, md5 76a24b4d…) — 11,175,629 lines, 1,577,982,906 raw bytes; fetched at bench time on the baseline host, md5-verified, never redistributed. Hardware. baseline-8vcpu-32gib — the §1 baseline (8 dedicated vCPU, 32 GiB RAM, local SSD), provisioned for this run and torn down immediately after. These are the authoritative C1 / C2 numbers for RFC 0001’s validated gates. Run. Dedicated baseline host (no CI run id): one ourios-bench --gates c1,c2 --hardware-kind baseline-8vcpu-32gib run at git 9a57ace; results JSON retained by the maintainer (2026-06-14T00-36-23.225Z-9a57ace.json). A1 was deliberately not run — it is diagnostic, not gating (RFC 0011); the §9.5 diagnostic A1 reading stands.

Authoritative confirmation. The verdicts match §9.5’s local diagnostic run bit-for-bit — expected, since C1 (reconstruction fidelity) and C2 (template-count convergence) are deterministic functions of (corpus, miner) with no wall-clock or hardware-sensitive component (contrast A1’s writer-environment sensitivity, §9.4). The value of this run is the authoritative hardware_kind stamp on the two gates that, under RFC 0011, define RFC 0001’s validated: both PASS on a representative ≥ 1 M-line corpus on §1 baseline hardware.

9.7 Results — 2026-06-15 (authoritative, baseline-8vcpu-32gib) — D2 / D3 / B2-post (RFC 0009 compaction)

Hardware. baseline-8vcpu-32gib — the §1 baseline (8 dedicated vCPU, 32 GiB RAM, local SSD), provisioned for this run and torn down immediately after. These are the authoritative D2 / D3 / B2-post numbers for RFC 0009’s validated measure (RFC0009.7). Run. Dedicated baseline host (no CI run id): the ourios-bench compaction bench at git 4d52288. Two invocations — the band-scale one-shot (OURIOS_COMPACTION_BASELINE=1, FILES=32, ROWS=4800, BODY_BYTES=4096) for D2/D3, then the b2-post-compaction criterion group. Synthetic (no corpus): D2/D3 drive one partition of 32 small files (~485 MiB) through compact_partition; B2-post queries 32-files-vs-1-file with the result set held constant.

Reading. D3 is the headline: a band-scale compaction lands its output squarely in the H4 256 MiB–2 GiB target with zero sub-128 MiB files — the small-file problem, eliminated. D2 shows consolidation runs at ~167 MiB/s on one partition/thread, far above any plausible per-partition seal rate, so a backlog drains (the “keeps up” property). B2-post quantifies the query payoff that motivated RFC 0009 (the PR #92 B2 finding that per-file footer/metadata reads dominate): collapsing 32 files → 1 cuts the footer reads ~6× on this query. The structural reductions (32 → 1 files / row groups, rows conserved) are hardware-independent and also pinned in ourios-parquet’s rfc0009_1_* / compaction_conserves_every_row tests; these wall-clock figures are the baseline-hardware stamp for RFC 0009’s validated. The full sustained-ingest soak (D2’s “backlog returns to zero in a one-hour window at D1’s rate”) and D1 itself remain unrun — the throughput here is the RFC0009.7 D2 measure, not that soak.

9.8 Results — 2026-06-18 (authoritative, baseline-8vcpu-32gib) — ingest write-path + recovery (criterion) and real-corpus A1 / C1 / C2 + B1 / B2

Hardware. baseline-8vcpu-32gib — the §1 baseline (8 dedicated vCPU, 32 GiB RAM, local SSD), provisioned for this run and torn down immediately after. Two such hosts (one per invocation set), both at git d3f2cae. Run. (a) the self-contained ourios-bench criterion benches ingest_write_path (RFC 0014) and recovery (RFC0008.3) — synthetic, no corpus — at full criterion settings; (b) the ourios-bench binary --gates a1,c1,c2 against two real corpora, plus the b1/b2 criterion benches (--warm-up-time 1 --measurement-time 3, matching query-bench.yml) over those corpora. Corpora: LogHub HDFS_v1 (Zenodo record 8196385, md5 76a24b4d… — 11,175,629 lines / 1,577,982,906 raw bytes (1.47 GiB) of real Hadoop production logs, above §8’s ≥ 1 GiB canonical minimum) and the frozen OTel-Demo v1 (corpus/otel-demo-v1, 38,782 lines / 31.5 MiB). HDFS is fetched in-job and never redistributed (§1).

(a) Ingest write path + recovery — supportive wall-clock (criterion).

Single-threaded micro-benches on synthetic records — supportive wall-clock (the structural sides are pinned by ourios-ingester’s RFC 0014 / ourios-wal’s RFC0008.3 tests), not gates. Dedicated hardware ran ~20–30% faster with much lower variance than the indicative ci-runner figures.

(b) Thesis gates A1 / C1 / C2 on real corpora.

C1 reconstructs every non-lossy row bit-for-bit across 11 M real production lines — the §3.3 invariant holds on real data at scale. C2 converges on HDFS (40 templates over 11 M lines; ratio 0.825 ≥ the threshold) — the template-mining thesis on a real corpus. A1 fails as expected: it is a recorded diagnostic, not a gate (RFC 0011) — template mining’s value is query pruning (B1/B2), not on-disk bytes beating a whole-stream codec.

(c) Query gates B1 / B2 on real corpora.

The windowed HDFS arm is the headline: a time-bounded query on the real 11 M-line corpus prunes 13 of 14 row groups via Parquet min/max statistics — the predicate-pushdown thesis (pillar #1) on real production data, ~5× faster than the unwindowed scan. (B1’s real-corpus arm skipped: OTel-Demo v1 has no error-band severity_text rows for the selectivity probe.) B1/B2’s structural pruning is the gate (pinned in ourios-querier); these are the baseline-hardware wall-clock stamp.

Not committed by the bench tooling — this is the curated narrative; the managed BENCH-RESULTS region above is for --update-benchmarks-md runs. The b1/b2 criterion timings use the reduced --warm-up-time 1 --measurement-time 3 (matching query-bench.yml); the structural pruning/template numbers are exact and criterion-setting-independent.

9.9 Results — 2026-07-03 (indicative, ci-runner) — B1 / B2 post-RFC 0022 (promoted attribute columns)

Purpose. The RFC 0022 §5 RFC0022.5 note: the promoted-attribute write path (per-key resource.<k> / attr.<k> columns + the two-arm predicate compile) must leave B1/B2 unchanged. This is the indicative re-run after RFC 0022 went green (#345–#348); the pruning counters are pinned structurally in crates/ourios-querier/tests/rfc0022_attr_columns.rs, this entry is the wall-clock stamp. Corpus. corpus/otel-demo-v4 (107,332 records → 735,377 mined rows / 5 files) and corpus/otel-demo-v5 (163,929 records, ~1.04 GB raw → 1,367,532 mined rows / 6 files). The LogHub HDFS_v1 arm did not run (fetch_hdfs off — memory-bound on the hosted runner). Hardware. ci-runner — indicative, not the §1 baseline. Run. query-bench.yml 28686650566 at git 6e3301b (the RFC 0022 green merge).

B1 — predicate pushdown vs zstdcat | grep (target: ≥ 10× at 1 GiB). Query: severity ERROR, full corpus span.

Row count agrees exactly with the reference pipeline. v4 is skipped as in §9.3 (its capture has no error-band rows).

B1 verdict: PASS (indicative), no regression — 34.8× against §9.3’s 40.0× on the same corpus, comfortably inside hosted-runner noise and 3.5× above the bar. Same caveats as §9.3: ultra-thin error band, corpus just under the §8 minimum, not the §1 baseline.

B2 — template-exact latency ∝ result, not corpus.

B2 verdict: PASS (supportive, indicative), no regression — the windowed latencies sit in the same flat few-ms band as §9.3/§9.8 while the full-span variants grow with corpus size, and everything is orders of magnitude under the 200 ms bar. The formal target speaks above ~10 GiB, which remains unmeasured on this runner class.

Assessment. The promoted-column machinery (extra column chunks per row group on the write side; the two-arm OR compile on the read side) shows no measurable drag on either gate. The RFC 0022 green → validated step still requires the authoritative baseline-8vcpu-32gib rerun per the standing bench policy (maintainer opt-in); this entry is its indicative precursor, curated by hand as in §9.3 — the workflow never writes §9.

9.10 Results — 2026-07-04 (authoritative attempt, baseline-8vcpu-32gib) — B1/B2 at 16 GiB: run blocked, miner finding

Purpose. The first run in the §8 10–100 GiB band: B2’s formal target speaks above ~10 GiB and had never been measured there. Corpus. LogHub HDFS_v2 (bench-time fetch, never redistributed): 31 files, 17,240,888,465 bytes ≈ 16.1 GiB raw, ~71 M lines of Hadoop daemon logs — the first corpus in our set whose shape (stack traces, multi-format node logs) differs qualitatively from HDFS_v1’s block events. Hardware. baseline-8vcpu-32gib, provisioned for the run and torn down after. Outcome: the run did not complete — it produced a product finding instead. The B2 store build was OOM-killed at 31.5 GiB RSS: the miner mints templates without bound on this corpus shape (template ids ≥ 56,199 by the 1.8 GiB subset mark, busiest template covering 0.67 % of 8.37 M rows; memory ~linear at ≈2× corpus bytes). Two bench-side pathologies were found and fixed en route — the eager corpus load (#350, now streaming: 1.3 GiB flat over hours) and a quadratic harness snapshot capture (#351, ~400× store-build speedup; gdb stacks exonerate the miner’s CPU path). RFC 0023 (bounded template memory) is the response; its RFC0023.7 criterion is this exact run completing.

What did land before the kill (recorded as diagnostic):

B2’s shape — flat windowed latency, window-driven pruning — holds wherever memory allows; the fragmentation itself (56 k templates, busiest at 0.67 %) also means pillar #2’s logical reduction fails on this corpus shape, which is the same finding from the pruning side. B1 did not reach its arms (stopped before the reference build once the OOM trajectory was clear). No gate verdict is claimed from this entry; the §8-band verdict waits on RFC 0023 + the rerun.

9.11 Results — 2026-07-04 (authoritative, baseline-8vcpu-32gib) — B1 / B2 at 16 GiB + RFC0023.7

Purpose. The §8 10–100 GiB band’s first completed measurement (the §9.10 attempt OOM’d), doubling as RFC0023.7 (bounded mining must complete this exact corpus under 8 GiB peak RSS) and the first B1/B2 readings at ≥ 10 GiB — where B2’s formal target speaks. Corpus. LogHub HDFS_v2 (bench-time fetch): 31 files, 17,240,888,465 bytes ≈ 16.1 GiB, 71,116,785 mined rows → 21 files / 80 row groups. B2 ran under the §3.3 Fixed severity baseline; B1 under the opt-in OURIOS_CORPUS_SEVERITY=log4j extraction (#350), stated per the methodology rule. Hardware. baseline-8vcpu-32gib, provisioned for the run, torn down after. Git 19e0886 (RFC 0023 bounds + telemetry merged).

RFC0023.7 — bounded mining at scale: PASS. Peak RSS 1.73 GiB across both benches’ store builds (5 s sampler), vs the §9.10 OOM at 31.5 GiB on identical input — an 18× reduction, under the 8 GiB bar with 4.6× headroom. Both benches completed (B2 phase 35 min; B1 including its zstd-19 reference build ~2.8 h).

B1 — predicate pushdown vs zstdcat | grep (target: ≥ 10× at 1 GiB, widening to ≥ 100× at 100 GiB). Query: severity ERROR, full 16 GiB span.

Row count agrees exactly with the reference pipeline.

B1 verdict: PASS (authoritative) — the ≥ 100× mark projected for 100 GiB is crossed at 16 GiB. With §9.8’s ~35–40× at ~1 GiB, the measured trajectory confirms the widening the target predicted: the reference’s cost grows with corpus bytes while Ourios’s grows with the matching row groups.

B2 — template-exact latency ∝ result, not corpus (formal target: ≥ 10 GiB, ≤ 200 ms for 10 k rows).

B2 verdict: PASS (authoritative, first ≥ 10 GiB reading) — the windowed query answers in the same few-ms band as the ~1 GiB corpora (§9.3/§9.8/§9.9): latency tracks the result, not the 71 M-row corpus.

The fragmentation datum (§9.10’s open question, quantified). The “busiest template” is id 0 — NO_TEMPLATE: under the default 20 k ceiling, ~79 % of HDFS_v2’s rows took the §6.3 parse-failure path (bodies retained bit-faithfully; observable via ourios.miner.parse_failure.reason, RFC0023.6). Template mining contributes little on this corpus shape — and the B1/B2 numbers above show the floor it degrades to (first-class-column + time pruning over Parquet statistics) still clears every gate. Follow-up noted: the B2 bench’s busiest-template picker should exclude NO_TEMPLATE so the full-span arm measures a true template-exact query on such corpora.

Assessment. RFC 0023’s §5 is fully discharged (this entry is the .7 record); the RFC flips red → green alongside this entry. The §8-band thesis verdict on real, hostile-shaped production logs: pruning compounds with scale (B1), result-bound latency holds (B2), and the mining-fragmentation failure mode is now bounded, observable, and priced.

9.12 Results — 2026-07-09 (indicative, local M-series) — otel-demo v8 capture: C1 / C2

The run is dated 2026-07-09; its C2 verdict was re-scored under the per-service gate on 2026-07-10 (#444 / RFC 0006 §3.4.3), so the resolution dates below post-date the heading.

Corpus. corpus/otel-demo-v8 (published GitHub release): a 48-hour OTel-Demo 2.2.0 capture at 150 locust users with the adFailure + paymentFailure feature flags active — 690,355 OTLP LogsData batches / 4,948,596 log records / 2.96 GB uncompressed, the largest and most hostile real capture to date (deliberately injected failure modes, multi-service, long-horizon). Calibration manifest at testdata/calibration/otel-demo-v8.json (RFC 0024 §3.1).

C1 — bit-identical reconstruction: PASS, perfect. The corpus holds 4,948,596 records (the calibration manifest’s count); 17 of them (all kafka, 0.0003 %) took the §3.3 lossy-flag path with their bodies retained, and C1 = 1.000000 over the remaining 4,948,579 rows — the honesty contract holds at 4.9 M rows through failure-mode churn.

C2 — template-count convergence (bar: ratio ≥ 0.5 at 1 M lines, evaluated per service since #444): PASS. Under the per-service gate (RFC 0006 §3.4.3, amended 2026-07-10) the corpus passes: the only service that clears the 1 M-line evaluation floor is cart, which converges at ratio 1.000 with two templates. Every other service abstains for want of volume; the whole-corpus ratio (0.199, end template count 14,631, sample cadence 4,833) is retained below as a diagnostic — it is a category error to grade a multi-service corpus as one Drain stream (§3.4.3 rationale). The per-service decomposition (splitting on service.name and re-running the gates per service) localises the whole-corpus fragmentation completely:

The gate folds over the gated services (those ≥ 1 M lines): cart is the sole such service and it passes, so the corpus passes. cart clears the formal gate at 2.76 M lines with two templates; the smaller services abstain below the 1 M-line floor, so they are not graded — though their observed counts (1–17 templates over 0.5–1.0 M lines) sit at the same near-flat convergence. The kafka broker, also abstaining, is the outlier: it mints 14,608 templates on 2.8 % of the lines. Mechanism (measured): kafka’s cleaner logs emit 3-token lines whose third token is a unique offset-bearing path (Deleted log /tmp/kafka-logs/…/00000000000000000429.log.deleted., 11,651 distinct) — one varying token in a 3-token line is similarity 2/3 ≈ 0.67, below the strict 0.7 threshold (§3.1 no-silent-merges), so each line mints a template; the 4-token siblings of the same family (0.75) merge fine. The failure-flag confound turned out to be a red herring. #444 settled how to handle the fragmentation (2026-07-10, maintainer-approved): of the three options — tokenizer masking, length-aware thresholding, and accept-and-scope-C2-per-service — option 3 shipped (the per-service gate, RFC 0006 §3.4.3, PR #451); masking is parked as a future strategic RFC (no commitment; a Collector transform or redaction processor can polish high-cardinality infra tokens upstream) and length-aware thresholding was rejected. The safety story held throughout (bounded memory per RFC 0023, per-service C1 perfect).

The per-service decomposition is now the first-class bench gate (ourios-bench --gates c2 prints it whenever any service bucket exists — distinct service.name values plus any <unknown>/<other>, so a single-service or plain-text corpus shows its one gated row too); template creation is a globally-monotonic event attributed to the minting service, so per-service creations partition the whole-corpus count exactly (2 + 17 + 1 + 3 + 14,608 = 14,631) in O(services) memory — no per-service id set. As of #444 (option 3) this decomposition is the gate: C2 is evaluated per service and folds over the services that clear the 1 M-line floor, with the whole-corpus ratio kept as a diagnostic (RFC 0006 §3.4.3).

What the fragmentation actually costs — B2 pricing (indicative, local M-series). Running the B2 windowed query on the fragmented (kafka) vs. converged (cart) service isolates the impact:

The deployed time/column pruning floor is identical whether a service has 2 templates or 14,608 — a 1 h window prunes 48 of 49 row groups either way (reconfirming the RFC 0023 graceful-degradation result on a fresh corpus). Fragmentation does not cost query latency or pruning. What it costs is template-exact query precision: probing cart’s dominant template (id 1 in this run — a run-specific identifier, not a canonical one) recovers 1.78 M / 2.76 M rows (one template is most of the corpus) but only 11,523 / 136,790 on kafka, because kafka’s dominant event is scattered across ~11,651 ids — a single template_id probe recovers only that one id’s slice (11,523 rows), not the full dominant event. So the fragmentation is a query-capability / thesis-value tradeoff, not a performance one; the pruning path degrades to the first-class-column floor unharmed. #444 accepted that tradeoff on hostile infra logs: the per-service gate makes C2 acceptance honest without masking, and any future masking is deferred to an upstream Collector processor or a dedicated RFC.

9.13 Results — 2026-07-12 (indicative, ci-runner) — RFC 0031 comparative program vs Grafana Loki (runs #8–#18)

Purpose. The first recorded numbers for the RFC 0031 comparative program — Ourios against Grafana Loki, the incumbent CLAUDE.md §1 defines the project against. These are the §7 calibration inputs the RFC’s open questions ask for, not gate verdicts: the L-gate margins are the RFC’s proposed values (M_L1..M_L4 = 10, F_L6 = 3, wired as ComparativeMargins::default()), the §5 gate scenarios (RFC0031.2–.11) are still red stubs, and the harness reports each pair under its provisional margin rather than asserting it. Every “PASS”/“fail” below is provisional pending the §7 freeze — a maintainer step; the open inputs are enumerated in point (4) of the closing Assessment.

Corpus. corpus/otel-demo-v8 (the §9.12 capture): 4,948,596 log records, 2.96 GB uncompressed — the RFC 0031 §3.3 headline corpus (real OTLP, failure flags active, kafka fragmentation and all). Both systems ingest the identical OTLP stream; an OTLP partialSuccess in any push response fails the run, so neither side can silently drop lines. Reference system. grafana/loki:3.5.3, digest-pinned (sha256:3165cecce301ce5b9b6e3530284b080934a05cd5cafac3d3d82edcb887b45ecd), single-binary mode, fed over its native OTLP endpoint. Flag deviations from stock are documented below — all ingest-replay accommodations, all in Loki’s favour, per the §3.7 anti-strawman commitment. Hardware. ci-runner — indicative, not the §1 baseline; the authoritative baseline-8vcpu-32gib run remains a maintainer opt-in per RFC 0031 §3.2. Bytes-read, the primary channel, is CPU-insensitive by construction, but nothing here is quoted as authoritative. Runs. comparative-bench.yml dispatch runs (curated by hand as ever — no workflow writes §9), each with one harness delta under test. Counted runs are equivalence-gated passes over the full corpus; the two diagnostic failures (#11/#13) are listed with exactly what they carry:

In every counted run, RFC0031.1 result-set equivalence held on every pair: the two systems’ answers, keyed (timestamp_unix_nanos, body_bytes), were multiset-identical at 4.9 M-record scale. Runs #11/#13 were L3-flicker diagnostics (an ingester-visibility artifact, fixed in #490 — see the deviations list); their table rows above note exactly what each carries. Every dispatched run appears in the table, and the per-class tables below carry a row for every run in each quoted streak (L1: #15/#16/#17; L3: #14/#15/#16/#17), so the streaks audit from this entry alone.

The metric (§3.6 as amended 2026-07-12). The Ourios figure is the total bytes fetched from object storage per query: count scan + row materialization + template-registry derivation. Loki is reported on two channels: storage-side (query-stats compressedBytes + headChunkBytes — the conservative apples-to-apples counterpart of Ourios’s fetched compressed-Parquet bytes; the harness evaluates gates primarily on this) and totalBytesProcessed (decompressed engine-side work, which overstates Loki’s storage reads by the chunk compression ratio; reported as context). Which channel the frozen §7 gates ride is an open maintainer decision.

Program history — the biased ruler, retired. Runs #5–#7 predate the §3.6 measurement-fidelity amendment and measured the Ourios side as the count scan alone (e.g. run #7’s severity figure of 609,498 B and its “146.9×”-style ratios), silently excluding the row-materialization and registry IO while Loki’s counterpart figure includes delivering results. Those runs are program history only and are not citable; every number below is on the honest total.

L1 — template-exact lookup (must-win, the flagship class): provisional PASS, widest margins. Pair: template_id == 4323 (2 rows) vs the LogQL line-filter needle "Updated connection-accept-rate max connection creation rate to" over every stream — the picker proves the two select identical row sets before the pair counts. Loki has no template concept, so its honest equivalent is a substring scan of the whole corpus; Ourios rides the writer’s existing bloom filter on template_id.

Above the provisional M_L1 = 10 on both channels, in every run since the pair landed (third consecutive pass at #17). The Loki side is structural: no template id → nothing to prune with.

L3 — trace correlation (must-win, OTLP-native): provisional PASS after blooms. Pair: every log line for one trace_id (9 rows). trace_id is high-cardinality by construction, so it cannot be a Loki label (§3.3’s machine-checked disallowlist); Loki’s honest query is a structured-metadata filter over all streams.

Run #12 is the honest before-picture: without blooms Ourios itself had to fetch the trace_id column corpus-wide, and the storage-side ratio (1.41×) was nowhere near the margin. The blooms (implemented in #489; the RFC 0005 §3.6 amendment recording them, with this as its measured evidence, is #491) collapse the fetch 15×, and the pair has now passed the provisional margin on both channels three runs in a row. As with L1, Loki’s side is structural: a trace cannot be pre-narrowed to a label stream, so it scans and decompresses everything in the window.

L2 — severity predicate (must-win family): parity-plus storage-side, ~33× processed — not a provisional 10× pass. Pair: lowest-volume single-severity_text band on the highest-volume service, full corpus span, 1 row. The run series doubles as the read-path optimisation ledger (component split: count scan + materialize + registry):

(Run #8’s Loki side: 2,880,784 storage / 89,184,711 processed.) Across the later reproductions the storage-side ratio sits at 1.05–1.31× and processed at ~33–39×, the spread being entirely Loki-side wobble (below). Reading: on the honest metric Ourios went from losing the storage channel (0.67×) to parity-plus via two read-path fixes, and wins decisively on engine work — but this is not a 10× storage-side pass, and no amount of wobble makes it one. The remaining named levers: the constant 513,862 B template-registry derivation, 20–29 % of every small-answer query’s total (the RFC 0033 cached-template-map candidate), and write-side page/row-group sizing.

Time-window browses (L6 floor family): published loss on the storage channel. Pairs: all lines of the highest-volume service in a clean k-row window (the promoted-column bloom’s worst case), plus run #17’s diagnostic — the same shape scoped to the lowest-volume service (“ad”, ~34 s window), where the service.name bloom could in principle skip. Floor gate as reported here: a bytes-read floor analog (Ourios ≤ 3× Loki, i.e. ratio ≥ 0.33) — the harness applies the §7 F_L6 factor to this entry’s bytes channels. Note the §5 gate as written (RFC0031.7) defines the L6 floor on latency p50 — measured in run #18 (see the latency section below), where the gate as written passes on all three window pairs; the bytes framing here remains the conservative reporting channel pending the §7 freeze.

This is the honest loss the RFC’s L6 disposition anticipated, and it is published as §5 RFC0031.11 demands: on a browse-k-rows query Loki reads only the tiny chunk slice its label stream + time index point at, while Ourios pays fixed per-query costs (the registry constant plus row-group-granularity materialization) that dwarf a k-row answer. The #486 late-materialization fix halved the loss and lifted k=2000 past the processed floor; storage-side stays 0.007–0.018 vs the 0.33 floor on current code. Run #17’s diagnostic sharpens the why: scoping to a low-volume service improves Ourios only ~22 % and flips the processed floor to pass, but there is no bloom collapse — v8’s hour partitions each hold roughly one row group containing all services, so the promoted service.name bloom has nothing to skip. The tier-changing lever is write-side layout (service clustering / row-group sizing — hazard #4 territory, an RFC-level change), not query-side tuning.

Latency (§3.6 channel, run #18 — the program’s first). Median of 7 warm repetitions per pair per system, measured only on correctness-verified pairs; Ourios timed in-process, Loki over localhost HTTP (negligible at these magnitudes; stated because latency is corroborating, not sole-gating):

Two findings this channel settles. First, the young-engine latency risk the RFC hedged against (“a latency loss + bytes-read win = sound architecture, young implementation”) did not materialize: Ourios answers every pair in 39–86 ms — a flat, fixed-cost-shaped profile — while Loki spans 13.8 ms to 24.1 s, and on the needle classes the wall-clock gap is interactive-vs-batch (75 ms vs 23–24 seconds). Second, scenario RFC0031.7 evaluated as written — on latency — PASSES on all three window pairs (0.34, 3.43, 1.32, all ≥ 1/3 at F_L6 = 3), and Ourios is outright faster on two of the three; the storage-channel loss published above is real as a bytes statement, but the RFC’s own L6 gate holds the floor. Which channel the frozen L6 gate uses is part of the §7 decision.

Determinism note. For repeated measurements of the same build and configuration, Ourios’s bytes are byte-identical (the store build is deterministic) — differences between runs are exactly the harness/optimisation deltas the table names, which is what lets the run series read as an optimisation ledger. Loki’s storage-side figure wobbles run to run (severity pair: 2.67–3.35 MB) with chunk boundaries and flush timing; ratios quoted against Loki carry that band.

Documented Loki flag deviations (all in Loki’s favour, per §3.7). The committed harness starts Loki with, and comments, exactly these deviations from stock:

• -validation.reject-old-samples=false — the frozen corpus is weeks old; stock Loki would reject the replay outright.
• -querier.query-ingesters-within=0 — stock Loki (default 3 h) skips ingesters for queries over weeks-old ranges, making rows still in unflushed low-volume chunks invisible (the run #11/#13 L3 flicker; diagnosed via ingester.totalReached: 0, fixed in #490). Disabling the cutoff means ingesters are always consulted — without it Loki’s answer to an old-range query is silently incomplete.
• Raised ingestion + per-stream rate limits (-distributor.ingestion-rate-limit-mb=512, -distributor.ingestion-burst-size-mb=1024, -ingester.per-stream-rate-limit=512MB, -ingester.per-stream-rate-limit-burst=1GB) — replay is far faster than the capture’s real-time rate.
• Raised internal gRPC message caps (-server.grpc-max-recv-msg-size-bytes=16777216, -server.grpc-max-send-msg-size-bytes=16777216) — runs #2–#4 failed on the same ~5.27 MB internal message regardless of our outer batch size: a single kafka-service LogsData line’s content alone inflates past Loki’s stock 4 MiB internal cap. Raising it (standard operator tuning) lets Loki accept the data at all, preserving the identical-ingest precondition the equivalence check requires.

Assessment. (1) The two classes the thesis stakes itself on hardest — L1 template lookup and L3 trace correlation — pass their provisional must-win margins on both channels, reproduced across three consecutive runs, and in both cases Loki’s cost is structural rather than tuning: no template concept, and no way to index a trace id. (2) L2 is parity-plus on storage and a ~33× processed win, honestly short of a 10× storage claim, with two named levers still on the table. (3) The window browses are a published storage-channel loss whose mechanism is understood (fixed per-query costs vs v8’s one-row-group-per-hour layout); the lever is write-side and RFC-sized. (4) Nothing here is frozen: the §7 inputs — the primary metric channel (storage-side vs processed), the must-win margins and floor factors, and whether the time-window pairs reclassify from gated floor to diagnostic — are open maintainer decisions, and this entry is the calibration evidence for them, not their resolution.

9.14 Results — 2026-07-13 (indicative, ci-runner) — comparative run #20: frozen gates on main, RFC 0033 acquisition

First dispatch on main after the §7 partial freeze and after the RFC 0033 cached template map merged (#511–#513). Job: run #20 (29255000054), exit 0.

Frozen gates. All asserting gates pass on main — M_L1/M_L3 storage margins and the F_L6 latency floors held; equivalence held on every pair. The dispatch is functioning as the regression gate the freeze intended (run #19 proved it on the branch; this run proves it on main).

RFC 0033 acquisition (the run’s purpose). Every pair reports:

template-map acquisition (RFC 0033): cold (audit fold, 513862 B; no artifact published)

• The registry component is byte-identical to run #8’s baseline (513,862 B constant per body-rendering query): the cache regressed nothing, exactly as the advisory design promised.
• But the write-through never published on this corpus, so no pair ever ran warm and the RFC0033.6 corpus gate (warm/cold ≤ 1/10) could not be measured.
• The explanation consistent with the run’s outputs is §3.2’s size abstention: the artifact is uncompressed JSON carrying every (template_id, version) canonical template string, while the 513,862 B it must undercut is zstd-compressed Parquet of the same strings (plus their event history). On v8’s template set the JSON evidently meets or exceeds the fold, and the guard refuses a publish that would make warm acquisition cost more bytes than the fold it replaces. (A publish IO failure would leave the same “no artifact” label; the §3.7 publish-outcome telemetry distinguishes the two in a served process, but the bench harness does not export metrics — the amendment run should print the outcome explicitly.)

Consequences recorded.

1. RFC 0033 status reverted green → red (this PR): RFC0033.6’s corpus arm is undischarged. The local-shape arm (55.8× on the 64-event fixture) stands.
2. M_L2 stays frozen-deferred — §7’s unfreeze condition (the RFC 0033 warm measurement on the headline corpus) was not met.
3. The lever is an artifact encoding amendment (format_version 2, compressed body). The same template strings zstd-compress into the 513,862 B audit Parquet with full event history alongside, so a compressed artifact is expected to land well below the fold size — to be measured, not assumed. Abstention semantics stay: publish only when the artifact beats the fold.

Roadmap to MVP

Living document. Refreshed at phase boundaries (§4) and whenever a merged PR materially changes the current state in §3. Last updated: 2026-06-15 — RFC 0013 (object storage, S3-compatible) drafted → specified → red (first shipping-milestone spine; store module skeleton + §5 stubs landed); RFC 0009 (background compaction) flipped to validated (RFC0009.7 D2/D3/B2-post measured on baseline-8vcpu-32gib, §9.7); RFC 0005 (Parquet storage) and RFC 0010 (audit-stream / drift queries) flipped to green (RFC0005.6 row-group sizing landed; RFC 0010’s eight §5 drift scenarios all pass). Earlier, on 2026-06-14, RFC 0001, RFC 0008, and RFC 0011 flipped to accepted (maintainer sign-off). RFC 0001 reached validated first (C1/C2 pass authoritatively on the benchmarks.md §1 baseline hardware, §9.6; A1 is diagnostic per RFC 0011); RFC 0008’s validated is vacuous (no thesis gate); RFC 0011 is a tuning RFC. The §§4+ phase narrative below predates this and is not re-verified here (PR #41 RFC 0005, then PR-D through PR-G landed ourios-parquet end-to-end: schemas, writer, reader, audit stream). The deferred-capabilities table in §5 is unchanged: WAL durability and the OTLP wire endpoints stay post-MVP.

This document answers two questions in one place: what does “MVP” mean for Ourios, and how far are we from it. The artifact is parallel to hazards.md and benchmarks.md: hazards say what we mustn’t break, benchmarks say what success looks like, and this file says how we get from here to there.

1. What “MVP” means here

MVP for Ourios is thesis-proving, not production-ready.

The thesis (CLAUDE.md §2) claims that Parquet + Drain-derived template mining + DataFusion collapses the inverted index, the compression layer, the storage tier, and the query engine into one stack of off-the-shelf parts plus thin glue. That claim is falsifiable. The MVP is the smallest stack that lets us run the thesis-gate benchmarks in benchmarks.md on a real corpus and either confirm the claim or kill it.

Production-shape concerns — gRPC OTLP receiver, WAL durability, snapshot mechanism, Helm chart, the full §6.8 telemetry surface, the RFC 0002 query DSL — are deliberately out of MVP scope (§5). Each is a real shipping concern, but none of them changes the answer to “does the thesis hold.” We defer to keep the critical path as short and honest as possible.

2. The MVP gate: thesis benchmarks

Four gating [THESIS] goals in benchmarks.md define MVP-done. Hitting all four on a representative corpus means the thesis holds; missing any of them means a pillar (CLAUDE.md §2) is wrong and a PR won’t fix it — an RFC will.

A1 (end-to-end compression vs. zstd-alone) was a fifth gating goal, but RFC 0011 (accepted) demoted it to a recorded diagnostic: it is refuted on every corpus class — including the maximally-templated one — for structural reasons (the more templated a corpus, the more a whole-stream byte codec captures the same redundancy), so template mining’s compression value is logical / query-pruning, captured by B1/B2, not on-disk bytes vs a codec. A1 is still measured and recorded (benchmarks.md §7/§9 — the columnar queryability premium + a codec-regression guard) but does not block MVP-done or any RFC’s validated.

A2, B3, C3, C4, D*, E* in benchmarks.md are relevant but not MVP-blocking — they’re tuning goals, honesty goals, or post-MVP shipping concerns.

3. Current state (as of 2026-06-15)

The thesis is proven on representative corpora. All four gating thesis-gates pass authoritatively on the benchmarks.md §1 baseline hardware (the §9.4 / §9.6 runs), so the MVP thesis-proving bar (§2) is met:

A1 (compression vs zstd) fails, but RFC 0011 (accepted) reclassified it a recorded diagnostic, not a gate: the failure is structural and template mining’s value is logical / query-pruning, captured by B1/B2 (see benchmarks.md §2 / §7).

RFC ladder status:

Crates — all ten product crates are implemented (ourios-core, -miner, -wal, -parquet, -ingester, -querier, -server, -bench, -semconv, -telemetry):

• ourios-miner — the Drain-derived miner, RFC 0001 accepted: (severity, scope) keying, three-zone confidence, widening + type-expansion with audit events, 256 B param-overflow spill, bit-identical reconstruction + the H7.3 render contract, structured-body canonical encoding, and §6.9 snapshot + v2 restore. Zero #[ignore]/todo!() acceptance stubs.
• ourios-wal — RFC 0008 accepted: append/sync, crash recovery (the real-SIGKILL CI gate), snapshot-restore, segment rotation, group-commit batched fsync, checkpoint-driven truncation; §5 arms .1–.10 green.
• ourios-parquet — RFC 0005 §3: atomic-publish writer + reader with the §3.9 compat contract, the §3.7 audit-event series, and the §3.6 encoding policy (dict + page index + template_id bloom filter).
• ourios-ingester — RFC 0003 green: the OTLP gRPC + HTTP receiver with WAL-before-ack, per-ResourceLogs tenant derivation, the windowed group-commit coordinator, and the startup recovery driver; also hosts the RFC 0009 compaction runner.
• ourios-querier — RFC 0007 validated / RFC 0002 green: the logs DSL over DataFusion with predicate + partition (time-window) pruning, alias resolution, and the RFC 0010 drift query.
• ourios-bench — RFC 0006 green: drives the A1/B1/B2/C1/C2 measurements over OTLP-Demo + LogHub corpora and records results to benchmarks.md §9.
• ourios-core / -semconv / -telemetry / -server — shared types + tenancy + record/audit shapes; the weaver-generated OTel name constants; the OTel metrics/export surface; the two-role binary.

The full cargo test --all-features suite is green in CI — the cargo test job gates every PR on the exact head; the coverage job runs alongside it but is informational (continue-on-error), not gating.

What remains is post-MVP shipping shape (§5 — Helm chart, the production deployment surface) and the items tracked in the RFCs’ §7/§9 open-questions (e.g. RFC 0009’s full D2 sustained-ingest soak + a measured D1, and the S3 atomic-swap primitive). RFC 0005 (green) and RFC 0009 (validated) are no longer open.

4. Path to MVP — three phases

Phase scope only; per-PR breakdown lives in the planning that opens each phase, not in this doc, so the file stays stable as mid-stream design decisions land.

Phase 1 — Finish the miner

Goal: the miner mines, audits, retains bodies, reconstructs. By the end of this phase the miner self-contained covers RFC 0001 §6.2 / §6.3 / §6.4 / §6.5 / §6.6 end-to-end and most §5 scenarios are green.

Capabilities to land:

• Drain tree (root → length-N nodes → prefix nodes → leaves) with descend.
• Best-candidate selection in MinerCluster::ingest via sim_seq (replaces the exact-match HashMap placeholder).
• widen step + template_widened audit emission + type-expansion + template_type_expanded audit + degenerate- template guard.
• Three-zone confidence branching (clean / lossy / parse-failure)
  • body retention in the lossy zone.
• Separators preservation through the ingest pipeline + reconstruct() + lossy_flag semantics per §6.6.
• Per-parameter byte-limit check + OVERFLOW marker + forced body retention.
• MinerCluster::ingest consumes a structured OtlpLogRecord (per RFC 0001 §6.1 as amended), not a raw &str. The body_kind = String / body_kind = Structured fork lands with the §6.2 algorithm rewrite (a follow-on PR to the §6.1 amendment). Severity, scope, and the OTLP-canonical JSON encoding for structured bodies all flow through the miner from this phase forward.

Unblocks: thesis gates C1 (reconstruction) and C2 (template-count convergence). RFC 0001 §5 scenarios H1.*, H2.*, H5.*, H7.*, §3.3.1, RFC0001.* should mostly flip in this phase.

Phase 2 — Records to Parquet

Goal: mined records become Parquet files. By the end of this phase a corpus run produces on-disk Parquet that any DataFusion-aware reader can open.

Capabilities to land:

• New crate ourios-parquet.
• Record schema matching the amended RFC 0001 §6.1: identity + partitioning columns, the OTLP-derived columns (time_unix_nano, severity_number + severity_text, scope_name + scope_version, attributes, resource_attributes, trace_id + span_id + flags, event_name, dropped_attributes_count), and the body / miner-derived columns (body_kind, body?, params, separators, confidence, lossy_flag).
• Writer: record batch → Parquet file (with row-group sizing from hazards.md H4 — target 128 MB–1 GB row groups).
• Reader: Parquet file → record batch (for verification + the Phase 3 DataFusion path).
• Audit-event Parquet stream (the contract called out in RFC 0001 §9 “Cross-RFC contracts pending”).

Unblocks: thesis gate A1 (compression ratio). The Parquet column codec earns its share of the 50–200× headline only once records actually land on disk in this format.

Out of MVP scope, parked here: background compaction (small-file problem, hazards.md H4) — corpus runs are bounded, a single Parquet file per phase is acceptable; production compaction is a post-MVP PR.

Phase 3 — DataFusion + bench

Goal: the thesis-gate benchmarks run.

Capabilities to land:

• New crate ourios-querier — register the Phase 2 Parquet files with DataFusion and accept raw SQL. No DSL — RFC 0002’s surface is a post-MVP concern; the bench can use SQL directly.
• New crate ourios-bench — corpus runner that reads pre-recorded OTLP LogsData test data into a stream of OtlpLogRecords, hands them to the miner, writes Parquet, runs the A1/B1/B2/C1/C2 measurements, and reports numbers that go into benchmarks.md §9 (Status). No network receiver in MVP — the bench reads OTLP from disk, not from a gRPC/HTTP listener (those stay post-MVP per §5).
• testdata/corpus/ — anonymised real-log corpus committed to the repo (or a download script if size demands), serialised as OTLP LogsData (canonical JSON or protobuf) so the bench exercises the same record shape an OTel deployment would produce.

Unblocks: thesis gates B1 (predicate-pushdown latency) and B2 (template-exact latency). At the end of this phase, benchmarks.md §7 (the thesis-gate summary) has measured numbers for every [THESIS] row, and either the thesis holds or it doesn’t.

5. Deliberately out of MVP

Each item is a real production concern. The reason it’s deferred is “answering ‘does the thesis hold?’ doesn’t require it,” not “we don’t think it matters.”

Note on OTLP scope. The pre-amendment roadmap listed “OTLP receiver (gRPC + HTTP)” as a single post-MVP item. PR #20 + #21 split that scope: the OTLP record shape (OtlpLogRecord consumption, the canonical JSON encoding, the OTLP-aligned Parquet schema) is in MVP — it’s a prerequisite for thesis-gate C2’s validity, because the template-count convergence the corpus measures has to be over records that look like real OTel traffic, not over flat-text caricatures of it. Only the wire endpoints — the actual gRPC/HTTP listeners that decode OTLP off the network — remain post-MVP, and that’s the row in the table above.

6. Update cadence

This file refreshes:

• After every merged PR that materially changes §3 (current state) — the merging PR’s author (or their drafting assistant) updates the table and the §5 scenario count.
• At phase boundaries (§4) — when Phase 1 finishes, §3’s current state and §4’s “blockers” tables are reconciled, and the next-phase opening planning PR is summarised here.
• When a thesis-gate result lands in benchmarks.md §9 — this doc gets a one-line note in §3 acknowledging the result.

The doc is intentionally not refreshed on every spec edit — RFC patches and hazards.md edits don’t change the road map unless they change what MVP requires. If you find yourself updating §3 every PR, the doc has become an activity log; the fix is to be more selective, not to stop updating.

RFCs

Referenced from CLAUDE.md §5.1. This document is the minimum viable RFC process for Ourios. It will grow as the project does.

When an RFC is required

Per CLAUDE.md §5.1, an RFC precedes implementation for any change that touches:

• An architectural pillar (CLAUDE.md §2).
• An invariant (CLAUDE.md §3).
• A hazard (CLAUDE.md §4 / docs/hazards.md).
• The on-disk Parquet schema (CLAUDE.md §3.5).
• A new crate (CLAUDE.md §7).

Bug fixes, dependency bumps, and internal refactors do not need RFCs. When in doubt, assume RFC.

File layout

• Filename: NNNN-short-kebab-title.md, e.g. 0001-template-miner.md.
• Numbers are assigned in merge order. Draft PRs may use the next free number provisionally; if two drafts collide, the later-merged one renumbers.
• One file per RFC. Supersessions are recorded in the frontmatter of both the old and new RFC.

Required frontmatter

---
rfc: NNNN
title: Short descriptive title
status: drafted | specified | red | green | validated | accepted | rejected | superseded
author: Name <email>
drafting-assistance: Claude   # omit if no LLM drafted
created: YYYY-MM-DD
supersedes: —                 # or RFC NNNN
superseded-by: —              # or RFC NNNN
---

The maturity stages (drafted through validated) are gates an RFC moves through before it becomes binding; accepted is the terminal post-maintainer-signoff state; rejected and superseded are the off-ramps. See docs/verification.md §3.

Required sections

Every RFC has at least:

1. Summary — 3–5 sentences. The commitment, not the rationale.
2. Motivation — why this change now, and why at this layer.
3. Proposed design — precise enough that two engineers would produce the same implementation.
4. Alternatives considered — one paragraph each. “I have not heard of it” is not acceptable.
5. Acceptance criteria — normative scenarios, one per invariant or hazard the RFC touches. Format: structured prose with Given / When / Then / And leading clauses; each scenario carries an id of the form H1.1, §3.4.2, or RFC<NNNN>.<m>, referenced from the test code so the mapping is greppable. See docs/verification.md §2.
6. Testing strategy — mapped to CLAUDE.md §6.2; references the §5 scenario ids and names the technique (proptest, corpus, criterion) for each.
7. Open questions — everything unresolved, as a checklist.
8. References — paper citations, related RFCs, CLAUDE.md sections constrained.

Additional sections are welcome when they clarify. Do not pad for the sake of the template.

Lifecycle

The five-stage maturity model. An RFC moves through these stages before becoming binding; the status: frontmatter field tracks the current stage so reviewers and tooling see it without reading the body.

1. Drafted — PR opened with status drafted. Sections §§1–4 and §§7–8 are filled. Discussion happens in PR review.
2. Specified — §5 acceptance criteria are written, every invariant and hazard the RFC touches has at least one scenario, and review has confirmed the criteria are testable in principle.
3. Red — test stubs exist and fail. Implementation may begin.
4. Green — all acceptance criteria pass; unit + property + corpus tests green.
5. Validated — thesis-gates in docs/benchmarks.md §7 pass on representative corpora. Maintainer flips status to accepted.

A regression detected after Validated either reopens the RFC (if a criterion is invalidated) or spawns a tuning RFC per benchmarks.md §7 (if a thesis-gate degrades). See docs/verification.md §3.

Two terminals reachable from any stage:

• Superseded — a later RFC replaces part or all of this one. Both frontmatters are updated. The superseded RFC is not deleted.
• Rejected — closed PR or status flipped to rejected. The file is kept for the record.

Diagrams

When an RFC needs a diagram (state machine, sequence flow, schema relationship, decision tree), it is authored in Mermaid, embedded as a fenced ```mermaid block in the markdown. Mermaid is chosen for the same reasons we chose markdown over a binary doc format: text-based source is reviewable in PR diffs, version-controllable, and lets the RFC itself remain a single self-contained file.

Lectures (docs/talks/) use a different convention: hand-drawn SVGs (Excalidraw export, or hand-authored to match) committed under docs/talks/img/. Lectures benefit from a “manuscript / blackboard” aesthetic that Mermaid does not provide; RFCs benefit from the diff-ability that Excalidraw does not provide. Do not mix the two conventions.

The mdBook build has the mdbook-mermaid preprocessor enabled (book.toml), with the Mermaid runtime vendored at the repo root (mermaid.min.js, mermaid-init.js) so the rendered book is self-contained. The CI book job and the Pages workflow install the mdbook-mermaid binary before building. To work on diagrams locally, cargo install mdbook-mermaid --locked (the preprocessor binary) — the vendored runtime is already committed.

Relationship to architecture docs

An accepted RFC is a contract for how something will be built. Once the subsystem is stable, the RFC graduates to docs/architecture/<subsystem>.md — a living document describing the system as it actually is. The RFC stays in place as the historical decision record; the architecture doc is what a new contributor reads first.

RFC 0001 — Template miner

rfc: 0001 title: Template miner (Drain-derived online log parsing) status: accepted author: Jens Holdgaard Pedersen jens@holdgaard.org drafting-assistance: Claude created: 2026-04-24 supersedes: — superseded-by: —

RFC 0001 — Template miner

Status note. accepted (2026-06-14, maintainer sign-off — the terminal ladder status per docs/rfcs/README.md). Reached validated the same day on the evidence below; accepted records the maintainer’s final sign-off on the template-mining pillar. The docs/verification.md §3 / docs/rfcs/README.md ladder reserves validated for every thesis-gate the RFC’s pillars touch passing on representative corpora (benchmarks.md §7). The template-mining pillar’s gates are C1 (reconstruction fidelity) and C2 (template-count convergence) — both pass on the representative LogHub HDFS_v1 corpus (≈ 1.47 GiB, 11.2 M lines — above benchmarks.md §8’s ≥ 1 GiB canonical floor, so representative; and well past C2’s ≥ 1 M-line formal-gate threshold, so that gate applies rather than abstains), authoritatively on the benchmarks.md §1 baseline hardware: C1 1.000000, C2 a 40-template plateau (diagnostic local run benchmarks.md §9.5, authoritative baseline-8vcpu-32gib rerun §9.6 — identical verdicts, as expected of deterministic gates). A1 (compression vs zstd) is a diagnostic, not a gate (RFC 0011, already encoded in the §7 gate table): it fails on every corpus class including the maximally-templated one, for structural reasons — template mining’s compression is logical/query-pruning (B1/B2), not on-disk bytes.

Reached green first (2026-06-13): all §5 acceptance criteria pass with live tests (zero #[ignore]/todo!() stubs) — miner-internal (tokenize/mask/sim-seq, the three-zone confidence model, fresh-leaf + widening + type-expansion with audit events, the (severity_number, scope_name) template key, the 256 B param-overflow spill + telemetry, bit-identical reconstruction + the H7.3 render contract, structured-body canonical encoding, the §6.9 snapshot + v2 restore) plus the relocated cross-crate criteria (query semantics RFC0001.5/.6, time-preserved RFC0001.10, §3.7.3 per-ResourceLogs tenant derivation, drift H5.3 via RFC 0010, the §6.7 alias index via RFC 0005 §3.7).

Terminal step: accepted is the maintainer’s final sign-off (docs/rfcs/README.md). NB the A1 re-scope’s own RFC (RFC 0011) is still drafted; the §7 gate table already reflects the demotion, but RFC 0011 should be accepted to fully ratify that chain.

How to read this document. §§1–4 are the design contract — the what and the why. §5 lists the normative Given / When / Then scenarios — the contract — grouped by parent (hazard, invariant, RFC-internal). §6 is the precise specification the ourios-miner crate is implemented against; its opening paragraphs name the gaps between the published algorithm and a production miner that §6.1–§6.9 then close. §7 records the alternatives we evaluated and rejected. §8 maps each §5 scenario to the technique that tests it.

Cross-references to CLAUDE.md sections are in square brackets, e.g. [§3.1], and name the invariant the section must preserve.

1. Summary

Ourios implements a Drain-derived online template miner (ourios-miner) that converts each ingested OTLP LogRecord into a structured Parquet record. The record shape is the OTLP LogRecord (with its inherited Resource and InstrumentationScope context) plus the miner-derived columns (template_id, template_version, params, separators, body_kind, body?, confidence, lossy_flag); see §6.1 for the full schema and the 2026-05-13 amendment that aligned it to OTLP. The miner is per-tenant by construction [§3.7], uses a three-zone confidence model that retains the original line in the lossy zone [§3.1], audits every template widening [§3.1], captures inter-token separators in a parallel array so that bit-identical reconstruction is the default rather than a property-test exception [§3.3], bounds parameter values at 256 B with overflow to a side body column [§3.2], and tracks template structural changes via a monotonic template_version so that schema drift across deploys is a first-class query rather than a silent count drop [§3.5]. The 50–200× figure is a logical reduction (lines → (template_id, params)), realised as query pruning (gates B1/B2), not on-disk bytes versus a byte codec — see RFC 0011, which demoted the compression-vs-zstd ratio (A1) to a diagnostic.

2. Motivation

This is the load-bearing pillar of the project [§2.2]. Three sub-questions justify it.

Why template mining at all. A typical service emits 10²–10⁴ distinct printf templates over its entire lifetime, but raw log volume is dominated by the parameters substituted into those templates. Storing the template once per tenant and the parameters per occurrence makes that redundancy explicit — and explicit redundancy stacks with byte-level codecs rather than fighting them. zstd over flat log text recovers ~10× on typical workloads; doing the structural work first leaves zstd a column of short, repetitive parameters that dictionary-encode well, where the codec then earns its keep again. The 50–200× headline (README.md, [§2.2]) is the product of these two layers, not a claim about either alone.

Why online vs. offline. Operators expect logs to be queryable within seconds of ingest, not minutes. Any batch clustering window long enough to do offline hierarchical clustering well is a window the operator is blind in. Drain’s fixed-depth tree gives O(d) lookup per line at the cost of being slightly less accurate than the best offline parsers — an acceptable trade because §3.1’s audit and confidence machinery surfaces the inaccuracy rather than hiding it.

Why this layer. The compression is structural, not statistical. Doing it before Parquet’s byte codecs means each Parquet column sees small, dictionary-friendly values; doing it after means we have already paid for storing the redundancy and zstd has to find it again from the bytes. The order matters.

3. Background: Drain as published

A restatement of He et al., ICWS 2017, in the notation this RFC uses downstream. Citations are by paper section/figure.

3.1 Tree structure (paper §3.2, Fig. 2)

A fixed-depth parse tree, depth d (default 4 in the paper). Three node kinds, in order from root:

• Root. Single node; routes by token count.
• Length-N node. One per observed token count N. Children are prefix nodes keyed by the first token of the line.
• Token-prefix nodes at depths 2..=d. Each is keyed by the token at position (depth - 1) of the line.
• Leaf log groups at depth d + 1. Each leaf holds a template — a sequence of N tokens, where each position is either a fixed string or the wildcard <*>.

                   root
                    │
        ┌───────────┼───────────┐
       len=4       len=5       len=6      ← length groups
        │           │           │
    ┌───┴───┐    ┌──┴──┐     ┌──┴──┐      ← prefix nodes (depth 1)
   "user"   …  "GET"  …    "INFO"  …
    │           │           │
   ┌┴┐         ┌┴┐         ┌┴┐
   …  …        …  …        …  …          ← prefix nodes (depth 2)
   │           │           │
 [leaf]      [leaf]      [leaf]           ← log groups

3.2 Similarity function (paper §3.3)

For a candidate line L = (t_1, …, t_N) and a leaf template T = (τ_1, …, τ_N):

simSeq(L, T) = (count of positions i where t_i == τ_i or τ_i is <*>) / N

Wildcards in the template count as matches. The line length and the template length are equal by construction (the length-N node selected the leaf candidates).

3.3 Threshold st (paper §3.4)

A configured value st ∈ (0, 1]. After computing simSeq against every leaf at the current parent, the leaf with the highest simSeq is the candidate. If simSeq(L, T_best) ≥ st, the line attaches to T_best; otherwise a new leaf is created. The paper reports st = 0.4 as a default; see §6.3 for why Ourios overrides this.

3.4 New-log-group creation vs. leaf update (paper §3.5)

If simSeq(L, T_best) < st, a new leaf is created at the parent prefix node, with L as its initial template (no wildcards yet). Otherwise T_best is updated: at every position where t_i ≠ τ_i, the template position is replaced with <*>. The template never becomes more specific over time, only more general; positions can become wildcards but cannot become fixed again.

3.5 Worked example

A fabricated illustration (no testdata/corpus/ exists yet; this example will be replaced with one drawn from the corpus once it lands).

Line A:  user 42 logged in from 10.0.0.1
Line B:  user 17 logged in from 10.0.0.2
Line C:  user 99 logged out from 10.0.0.7

After preprocessing (§4.2), numbers and IPs are masked, so the miner sees:

Line A:  user <NUM> logged in from <IP>
Line B:  user <NUM> logged in from <IP>
Line C:  user <NUM> logged out from <IP>

All three are length 6. They route to root → len=6 → "user".

A walks the prefix path further (depth 2: token at position 1 is the masked <NUM> placeholder, treated as a fixed token at this stage). It is the first line, so a leaf is created with template user <NUM> logged in from <IP>.

B walks the same path. The candidate leaf has simSeq(B, T_A) = 6/6 = 1.0 ≥ st. B attaches; the template is unchanged.

C walks the same path. The candidate leaf has simSeq(C, T_A) = 5/6 ≈ 0.833. With st = 0.7 (Ourios default, §6.3), 0.833 ≥ 0.7, so C attaches. Token position 4 (in vs out, 1-indexed against the masked sequence) becomes <*>. The template widens to user <NUM> logged <*> from <IP>. This is a template widening event and must emit an audit record per §6.4.

4. Background: Drain3 extensions (not in the paper)

Drain3 (logpai/Drain3) is the maintained Python implementation. It adds several capabilities beyond the 2017 paper. Each is recorded here as adopt, adopt with modification, or reject, with one sentence of rationale.

4.1 Persistent state — adopt with modification

Drain3 supports JSON snapshots to file, Redis, or Kafka. Ourios adopts the snapshot concept but commits to a file/object-storage backend; Redis and Kafka are out of scope (CLAUDE.md §3.6 names object storage as the source of truth). Snapshot target, cadence, and scope are open questions in §9.

4.2 Pre-tree-walk masking — adopt with modification

Drain3’s most important extension: regex-based masking of common parameter shapes (IPs, UUIDs, numbers, hex, timestamps, file paths) before the tree walk, so high-cardinality tokens never become tree branches. Without this, the tree explodes into one branch per IP address.

The Ourios modification: a masked token is not discarded. It becomes a typed parameter attached to the wildcard slot it created. The masking layer emits (type_tag, original_bytes) pairs; the tree walk treats the type tag as the token (so <NUM> matches <NUM> for tree-routing purposes) while the original_bytes flow into params so reconstruction can recover the line exactly. Paper-pure Drain loses the original token; Ourios retains it as a parameter. This is what makes [§3.3] reconstruction possible at all.

4.3 Variable-length wildcards — adopt with constraint

Drain3’s MaskingInstruction allows a single regex to match a variable-length run of tokens (e.g. a multi-token user-agent string). Ourios adopts this where the run is bounded at parse time and produces exactly one typed parameter in the output. Reject: unbounded variable-length wildcards, because they break leaf identity (two lines with the same template structure but different run lengths would land in different length-N nodes and never deduplicate).

4.4 Dynamic / adaptive threshold — reject

Drain3 supports auto-tuning the similarity threshold per leaf based on observed cluster sizes. Ourios rejects this. CLAUDE.md §3.1 fixes threshold ≥ 0.7 as a project-level invariant; auto-tuning would silently move the merge boundary across deploys, defeating the audit contract. Threshold tuning is a config decision per tenant, never a runtime decision per leaf.

4.5 Other Drain3 features

• Parameter-naming hints. Drain3 lets users name <*> slots via the masking config (e.g. <IP:client_addr>). adopt — the type-tag mechanism in §4.2 already requires a slot name; using the Drain3 hint format keeps configs portable.
• Built-in metrics surface. Drain3 exposes a set of state counters via callback. replace — Ourios exposes OTel metrics directly per §6.8 (instrumented via the meter API), with names that match [§3.1]’s required set rather than Drain3’s internal names.
• Parameter masking after the fact. Drain3 has utilities to retroactively mask params in already-clustered lines. reject — Ourios masks once, at ingest, deterministically. Retroactive masking would invalidate already-written Parquet files.

5. Acceptance criteria

Per docs/verification.md §§2–3, every CLAUDE.md §3 invariant and every docs/hazards.md hazard this RFC touches has at least one numbered scenario below. Scenarios use the bold-leading-clause format (verification.md §2.1) and the id grammars (§2.2): H<n>.<m> for hazard-rooted, §3.<n>.<m> for invariant-rooted, RFC0001.<m> for design-internal commitments. Test code carries each id in a doc comment per §2.3 so grep -R "H1.1" . resolves bidirectionally between RFC and tests.

The hazards in scope are H1, H2, H5, H7; the invariants are §3.1, §3.2, §3.3, §3.5, §3.7. H3 (WAL durability) and H4 (small files) are owned by the ourios-wal and ourios-parquet RFCs; H6 (DSL) is owned by RFC 0002; §3.4 (WAL-before-ack) and §3.6 (object-storage-as-truth) are touched only via §6.9’s persistence direction and the primary obligation lives in those other RFCs.

5.1 Hazards

Scenario H1.1 — Semantically distinct templates do not silently merge

• Given a corpus containing user logged in <*> and user logged out <*>
• When similarity threshold is 0.7 (the default)
• Then the two remain distinct template_ids
• And any widening produces an audit event recording both old and new templates

Scenario H1.2 — Lossy-zone match retains body

• Given a line whose best match has confidence in the lossy zone (floor ≤ x < threshold)
• When the line is ingested
• Then the body column contains the original line bytes
• And the row carries lossy_flag = false (the flag is reserved for tokenizer / preprocessing failure per §6.6 — the lossy zone retains the body but reconstruction still succeeds)

Scenario H1.3 — Every widening emits an audit event

• Given any sequence of inputs that triggers a template widening
• When the widening completes
• Then an audit event exists naming the old template, the new template, the tenant id, the timestamp, and the event_type

Scenario H1.4 — severity_number is part of the template key (no INFO/ERROR silent merge)

• Given two OtlpLogRecords with identical body_kind = String bodies and identical scope_name, but severity_number = 9 (INFO) and severity_number = 17 (ERROR)
• When both are ingested via MinerCluster::ingest
• Then the emitted records carry distinct template_ids
• And no widening or merge ever produces a single template_id covering both severity buckets
• (Operationalises the §6.1 Template-key composition commitment that severity_number is part of the key regardless of body_kind.)

Scenario H1.5 — scope_name is part of the template key (no cross-scope silent merge)

• Given two OtlpLogRecords with identical body_kind = String bodies and identical severity_number, but scope_name = Some("lib.auth") and scope_name = Some("lib.payments")
• When both are ingested
• Then the emitted records carry distinct template_ids
• And no widening or merge ever produces a single template_id covering both scopes
• And a third record with scope_name = None shares a template_id with neither (it lives in the (severity, None) bucket per §6.1)

Scenario H2.1 — Oversized parameter triggers OVERFLOW marker and forced body retention

• Given a tenant configured with the default 256 B per-parameter byte limit
• And a log line whose masked parameter value exceeds 256 B (e.g. an embedded stack trace)
• When the line is ingested
• Then the corresponding Param entry has type_tag = OVERFLOW carrying (length, sha256_prefix) instead of the original value
• And the body column contains the original line bytes regardless of lossy_flag
• And ourios.miner.params.overflow (attributes ourios.tenant, ourios.service) increments

Scenario H2.2 — Per-service overflow rate above 1% raises an alert

• Given the ourios.miner.params.overflow.utilization gauge (attributes ourios.tenant, ourios.service) for some service
• When the rolling rate exceeds 0.01
• Then the documented alert rule fires (the rule ships alongside §6.5’s metric definition)

Scenario H5.1 — Wildcard widening increments template_version and emits template_widened

• Given a leaf at (template_id = X, template_version = V)
• When an attach widens a previously-fixed token at position i into <*>
• Then the leaf’s template_version becomes V + 1
• And an audit event with event_type = template_widened is emitted naming the new wildcard position(s)

Scenario H5.2 — Type expansion increments template_version and emits template_type_expanded

• Given a leaf whose wildcard slot s has slot_types[s] = {NUM}
• When an attach maps a typed parameter of type = STR into slot s
• Then slot_types[s] becomes {NUM, STR}
• And template_version increments
• And an audit event with event_type = template_type_expanded is emitted naming the slot and the newly-added ParamType

Scenario H5.3 — Drift query returns templates that gained a version in window

• Given the template_audit event stream contains template_widened and template_type_expanded events for templates A and B in the window [t1, t2]
• When the §6.7 drift query runs against [t1, t2]
• Then the result includes both A and B with their widening counts

Scenario H7.1 — Reconstruction property holds across the corpus

• Given the committed testdata/corpus/ (anonymised, fixed)
• When every line is ingested through the miner
• Then for every emitted record r where r.lossy_flag = false, reconstruct(r) == r.ingested_bytes holds byte-for-byte
• And property failure is a build break, not a regression

Scenario H7.2 — Tokenizer failure sets lossy_flag = true and retains body

• Given a line containing an embedded NUL byte (or another tokenizer-failure mode listed in §6.6)
• When the line is ingested
• Then a parse-failure record is emitted
• And the record’s lossy_flag is true
• And the record’s body column contains the original line bytes

Scenario H7.3 — Reader emits body verbatim when lossy_flag is true

• Given a record with lossy_flag = true
• When the reader renders the row
• Then the rendered bytes are the body column verbatim (byte-for-byte, no prefix or in-band marker)
• And the rendered row carries the §6.6 reconstruction signal Reconstruction::RetainedVerbatim (the out-of-band warning marker the §6.6 Reader render contract defines)
• And reconstruct() is NOT called for that row

Scenario H7.4 — Widened literal slot reconstructs via STR fallback

• Given a leaf whose template gains a new <*> slot at position i via the §6.2 widening of an originally-literal token
• When the triggering line is attached
• Then the line’s record carries params[slot_for_i] = { type_tag: STR, value: L_tok[i] }
• And reconstruct(record) == ingested_bytes holds

5.2 Invariants

Scenario §3.1.1 — Default similarity threshold is 0.7

• Given a tenant configuration with no threshold override
• When the miner is initialised for that tenant
• Then the effective threshold is 0.7

Scenario §3.1.2 — Mandatory metric set is exposed

• Given the mandatory set defined by §6.8’s table — the ourios.miner.* metric instruments in the semconv registry (semconv/registry/, surfaced as generated ourios_semconv constants) that the miner registers on the ourios.miner meter when it is constructed (the ourios.miner.alias.* counters are registered separately by the alias map, §6.7, and are out of this scenario’s scope)
• When a small representative workload exercises every instrument (a normal line, a near-duplicate that widens a template, an oversized-param line, and a parse-failure line) and the meter is collected via an SDK in-memory reader
• Then the collected metric stream contains every metric named in §6.8’s table — each appearing on its first real measurement, carrying the registry’s required attributes (no synthetic zero-traffic points) — (ourios.miner.template.count, ourios.miner.merges, ourios.miner.confidence, ourios.miner.confidence.p50, ourios.miner.confidence.p01, ourios.miner.body_retention.utilization, ourios.miner.parse_failures, ourios.miner.params.overflow, ourios.miner.params.overflow.utilization, ourios.miner.template.version_changes, ourios.miner.duration) with the instrument kinds and attributes listed there

Scenario §3.2.1 — Default per-parameter byte limit is 256

• Given a tenant configuration with no per-parameter byte limit override
• When the miner is initialised for that tenant
• Then the effective limit is 256 bytes

Scenario §3.2.2 — Configured limit above 1 KiB is rejected at startup

• Given a tenant configuration with param_byte_limit > 1024
• When the miner is initialised
• Then initialisation fails with an error citing the §3.2 ceiling
• And the process refuses to start serving that tenant

Scenario §3.3.1 — Separators array captured on every successful tokenization

• Given a line that tokenizes successfully
• When the line is ingested
• Then the emitted record’s separators.len() == tokens.len() + 1
• And the per-row precondition for H7.1 holds (the reconstruction proptest then asserts byte equality)

Scenario §3.5.1 — Snapshot format carries a leading version byte

• Given a serialised snapshot artefact written by the miner
• When the artefact is inspected
• Then byte 0 is the snapshot format version

Scenario §3.5.2 — Unknown snapshot version triggers full WAL replay

• Given a snapshot artefact whose leading version byte is unknown to the running miner
• When the miner loads the snapshot at startup
• Then the snapshot is rejected
• And the miner falls back to full WAL replay rather than misinterpreting the bytes

Scenario §3.5.3 — Known-version restore + tail replay is equivalent to a full rebuild (2026-06-12 amendment)

• Given a tenant tree snapshotted at WAL high-water mark S, with further frames appended after S
• When recovery restores the snapshot and replays only the frames above S
• Then the recovered tree state (leaves, template_ids, template_versions, slot types, structured-template-id map) equals the control tree built by ingesting every record from scratch
• And no frame at or below S reaches the miner (no double-apply — the v1 hazard that gated restore)

Scenario §3.5.4 — Stale snapshot degrades loudly, not silently (2026-06-12 amendment)

• Given a snapshot at high-water mark S, a Parquet checkpoint at X > S, and a WAL whose surviving segments start above S but retain every frame above X (externally truncated — WAL segment files manually unlinked; the RFC 0008 §6.7 retain floor prevents this arising internally, and legitimate housekeeping never removes a frame above X)
• When recovery runs
• Then the snapshot is restored and the surviving frames are replayed under the per-consumer horizons (the data side is complete: every missing frame was ≤ X, hence already in Parquet)
• And a structured warning is emitted naming the gap between S and the oldest surviving frame, so the possible template re-minting inside it is surfaced (hazard #5, observable via the RFC 0010 drift query) rather than silent

Scenario §3.7.1 — Tenants’ template trees never cross-pollinate

• Given a MinerCluster ingesting interleaved lines from synthetic tenants A and B
• When the corpus is fully ingested
• Then no template mined under tenant A appears in tenant B’s tree
• And no template mined under tenant B appears in tenant A’s tree
• (Implements docs/benchmarks.md E2.)

Scenario §3.7.2 — Same structural template in two tenants gets distinct template_ids

• Given tenants A and B independently emit the structurally identical template user <NUM> logged in from <IP>
• When both are ingested
• Then tenant A’s template_id for that template differs from tenant B’s template_id
• And no template_id is shared across tenants
• And template_ids are guaranteed unique across the entire cluster (not just per tenant)

Scenario §3.7.3 — Tenant derivation runs per ResourceLogs, not per export batch

• Given a single OTLP ExportLogsServiceRequest carrying two ResourceLogs whose Resource.attributes resolve to distinct tenants A and B under the configured derivation rule
• When the receiver fans the batch out per RFC 0003 §6.3 and the miner ingests both per-tenant streams
• Then every LogRecord under ResourceLogs[0] is mined under tenant A
• And every LogRecord under ResourceLogs[1] is mined under tenant B
• And no record ever appears in the wrong tenant’s tree
• (Operationalises the §6.1 Tenant derivation commitment that the derivation rule runs once per inherited Resource, not once per export batch.)

5.3 RFC-internal design commitments

Scenario RFC0001.1 — Fresh-leaf creation does not emit an audit event

• Given a parent prefix node with no leaves yet
• When a line creates the first leaf at that node
• Then no event is appended to the audit stream for that creation
• And ourios.miner.template.count increments to reflect the new leaf

Scenario RFC0001.2 — Degenerate-template guard rejects fully-wildcard widening

• Given a leaf whose template, after a candidate widening, would have zero non-wildcard tokens
• When the candidate widening is attempted
• Then the widening is rejected
• And the line is treated as a parse failure (confidence = 0, body retained, ourios.miner.parse_failures increments)
• And an audit event with event_type = template_widening_rejected_degenerate records the rejection

Scenario RFC0001.3 — Tokenizer is Unicode whitespace only; punctuation stays in tokens

• Given a line key=value, other=42 (no whitespace adjacent to the punctuation)
• When the line is tokenized
• Then it produces two tokens (key=value, and other=42)
• And no token boundary is introduced at =, ,, :, ;, [, ], (, or )

Scenario RFC0001.4 — Confidence ratio = simSeq / threshold; decision boundary at 1.0

• Given a tenant with threshold = 0.7
• And a line whose simSeq against the best candidate is 0.7
• When the line is ingested
• Then the emitted record’s confidence == 1.0
• And the line takes the clean-attach branch
• And the same simSeq under threshold = 0.5 would yield confidence == 1.4 (the ratio reframes scale-invariantly across tenants)

Scenario RFC0001.5 — Bare template_id = X spans all versions of leaf X

• Given leaf X with versions 1, 2, 3 attached over time
• When a query runs where template_id = X
• Then the result includes rows attached against (X, 1), (X, 2), and (X, 3)
• And no alias resolution is involved (this is by-construction, since template_id is stable across widenings of one leaf)

Scenario RFC0001.6 — Bare template_id = X does NOT follow alias chains

• Given two distinct leaves X and Y that the alias index records as semantically equivalent
• When a query runs where template_id = X
• Then only rows whose template_id == X are returned; rows with template_id == Y are NOT included
• And where template_id.resolves_to(X) (RFC 0002 §5.4) is the explicit form that includes Y’s rows

Scenario RFC0001.7 — Combined widening + type-expansion increments version twice and emits two events in order

• Given a leaf at version V where a single attach both introduces a new wildcard slot AND introduces a previously-unseen ParamType into an existing slot
• When the attach completes
• Then the leaf’s template_version == V + 2
• And the audit stream contains two events for this attach: a template_widened event for the new wildcard, immediately followed by a template_type_expanded event for the type expansion (in that order)

Scenario RFC0001.8 — ourios.miner.confidence.p50 and ourios.miner.confidence.p01 are emitted as gauges

• Given a running miner with a non-empty ourios.miner.confidence histogram for some (ourios.tenant, ourios.service)
• When the miner’s meter is collected via an SDK in-memory reader
• Then ourios.miner.confidence.p50 and ourios.miner.confidence.p01 (attributes ourios.tenant, ourios.service) are present as gauges
• And each value matches the corresponding quantile of the same-attributed histogram (computed in-process on a short ticker per §6.8)

(The dotted-semconv rename landed in the 2026-06-08 amendment; the open fork is whether confidence.p50 / confidence.p01 become backend-derived quantiles over the exported histogram rather than in-process gauges. That is a contract change to the §3.1.2 mandatory set and would be made — possibly superseding this scenario — under its own review, not folded into the rename.)

Scenario RFC0001.9 — body_kind = Structured short-circuits to a structured-template id

• Given an OtlpLogRecord whose body is Body::Structured(AnyValue) (any non-String AnyValue variant carried verbatim per RFC 0003 §6.4)
• When the record is ingested
• Then the §6.2 algorithm skips tokenize/mask/descend per step 0 and allocates or reuses the structured-template id for (severity_number, scope_name, BodyKind::Structured)
• And the emitted record has body_kind = Structured
• And the emitted record’s body carries the Ourios canonical body encoding of that AnyValue (per the §6.1 encoding rule)
• And params and separators are empty
• And confidence == 1.0 (the §6.1 sentinel)
• And lossy_flag == false

Scenario RFC0001.10 — time_unix_nano is preserved verbatim from the wire

• Given an OtlpLogRecord with time_unix_nano = 1_715_700_000_000_000_000
• When the record is ingested and committed to Parquet
• Then the emitted row has time_unix_nano == 1_715_700_000_000_000_000
• And a query WHERE time_unix_nano BETWEEN 1_715_600_000_000_000_000 AND 1_715_800_000_000_000_000 returns the row
• (Gates docs/benchmarks.md B1 — time-range queries — by making the underlying column measurable.)

Scenario RFC0001.11 — severity_number = 0 and scope_name = None are distinct key buckets

• Given four OtlpLogRecords with identical body_kind = String body, varying only in (severity_number, scope_name) across (0, None), (0, Some("lib.x")), (9, None), (9, Some("lib.x"))
• When all four are ingested
• Then four distinct template_ids are emitted, one per key bucket
• And no widening or merge ever coalesces the severity_number = 0 (UNSPECIFIED) bucket with any specified-severity bucket
• And no widening or merge ever coalesces the scope_name = None bucket with any scope_name = Some(_) bucket
• (Locks the §6.1 explicit edge-case rules: 0 = UNSPECIFIED is a valid OTLP severity that gets its own bucket, and absent scope is its own bucket.)

Scenario RFC0001.12 — Alias assertion is durably recorded and appears in the per-tenant map

• Given tenant T with two distinct leaves A and B (A < B) and no existing alias set
• When an operator asserts B is an alias of A
• Then an alias_asserted audit event is durably recorded on the §6.4 stream under the §3.4 WAL-before-ack barrier before the assertion is acknowledged, naming tenant_id = T, the anchor representative_id = A, member_ids = [B], the actor, and the timestamp — so the asserted set is {A} ∪ {B} = {A, B}
• And after the per-tenant projection rebuilds, tenant T’s alias map contains an equivalence class with members {A, B} whose derived canonical representative is A (the smallest member)

Scenario RFC0001.13 — resolves_to(rep) returns all members and excludes non-members

• Given tenant T whose alias map records the set {A, B} (per RFC0001.12) and an unrelated leaf C in no set
• When the querier compiles template_id.resolves_to(A) for tenant T
• Then the predicate expands to template_id IN {A, B}
• And resolves_to(B) expands to the same {A, B} (expansion is by the set, not the direction of assertion)
• And resolves_to(C) expands to exactly {C}

Scenario RFC0001.14 — Cross-tenant isolation: an alias in tenant A never affects tenant B [§3.7]

• Given tenant T1 whose alias map records {A, B} and tenant T2 that has the same template_ids A and B but no alias assertion
• When the querier compiles template_id.resolves_to(A) once for T1 and once for T2
• Then for T1 it expands to {A, B}
• And for T2 it expands to exactly {A}
• (Locks §3.7: alias sets are per-tenant; an assertion in one tenant is invisible to every other.)

Scenario RFC0001.15 — Retraction removes any member, including the canonical, and is itself audited

• Given tenant T whose alias map records the equivalence class {A, B} (A < B, so A is the derived canonical)
• When an operator retracts member A — the canonical / smallest member — from the class
• Then an alias_retracted audit event is durably recorded (same WAL-before-ack barrier and field shape as RFC0001.12) whose asserted set names A (here as representative_id, the operator’s anchor) plus an empty member_ids, and the actor
• And after the projection rebuilds, A is removed from the class, leaving {B} — a single member, which is no longer an alias set, so resolves_to(A) expands to exactly {A} and resolves_to(B) expands to exactly {B}
• (Locks the representative-independent retraction rule: retracting any member is well-defined even when it is the canonical/smallest; the canonical is re-derived as min of the remainder, and a class that drops to one member ceases to be an alias set.)

Scenario RFC0001.16 — A non-aliased id resolves to itself

• Given tenant T with leaf Z and no alias assertion naming Z
• When the querier compiles template_id.resolves_to(Z)
• Then the predicate expands to exactly {Z} — identical to the base-member behaviour and to bare template_id = Z (RFC0001.6)

6. Proposed design

The Ourios miner in detail. This is the section that the ourios-miner crate is implemented against; §5’s Acceptance criteria operationalise the commitments here, and §8 maps each §5 scenario to the technique that tests it.

Why §6 exists. Published Drain (§3) and Drain3 (§4) do not address the properties Ourios requires. Each row below is a gap this section closes:

6.1 Data model

Amendment 2026-05-13. Section rewritten to align the record schema with the OTLP LogRecord shape — the project’s stated ingest contract per docs/glossary.md (entry OTLP: “we do not invent our own format”). The investigation that surfaced the gap is docs/architecture/otlp-log-format.md. The pre-amendment schema treated logs as raw text strings; the amended schema treats every log as a structured OTLP record from the moment it enters the system. §6.2’s algorithm and its ingest signature were aligned to this amendment in a companion edit the same day (see §6.2’s amendment note below): the body.kind fork is at the top of the algorithm, the descent step incorporates the §6.1 template-key tuple, and the MinerCluster::ingest signature now takes a structured OtlpLogRecord rather than a raw &str.

The miner emits one record per ingested OTLP LogRecord. The record shape mirrors the wire shape of OTLP logs (the opentelemetry-proto LogRecord plus its inherited Resource and InstrumentationScope context) plus the miner-derived columns that template mining produces.

Record columns

The record carries three groups of columns. The OTLP-derived group preserves the structured shape the wire promised; the miner-derived group is what this RFC introduces; the reconstruction group exists only when the body was mineable (body.kind = String).

Identity and partitioning:

OTLP-derived columns (faithful to opentelemetry-proto):

Amendment 2026-06-11 — the effective timestamp lives in RFC 0005, not here. RFC 0005 §3.2 (amendment of the same date) adds a writer-derived effective_time_unix_nano Parquet column — time_unix_nano when non-zero, else observed_time_unix_nano, else 0 — following the OTLP logs data model’s recommendation (“Use Timestamp if it is present, otherwise use ObservedTimestamp”). The record shape above is unchanged: the miner emits no new field, the Parquet writer computes the column from the two timestamp fields already listed, and the wire time_unix_nano is stored verbatim including 0 — scenario RFC0001.10 (verbatim preservation) remains intact and normative. Time partitioning and the DSL time window key off the derived column (RFC 0005 §3.4 / RFC 0002 §6.2).

Body and miner-derived reconstruction:

Where:

• Param = { type_tag: ParamType, value: Bytes }. ParamType is one of IP, UUID, NUM, HEX, TS, PATH, STR, OVERFLOW. STR is the unmasked-wildcard fallback — used when a slot was created by template widening of a previously-fixed literal token (the literal itself becomes the param value); OVERFLOW carries (length: u32, sha256_prefix: [u8; 8]) instead of the original value (§6.5). params.len() == count(<*> in template), always (in the body_kind = String branch); §6.2 enforces this when a widening introduces new wildcard slots.
• Separator is a small inline byte string (typically 1–3 bytes in practice). Encoding in Parquet is an implementation detail that does not affect this RFC.
• KeyValue mirrors the OTLP KeyValue message: a key: String and a value: AnyValue. AnyValue is a discriminated union over string | bool | int | double | bytes | array | kvlist. Storing AnyValue faithfully in Parquet (rather than flattening to a string) is what keeps query expressions like attributes["client.address"] = "10.0.0.1" typed.
• BodyKind is a two-variant enum (String, Structured) — not the full AnyValue discriminator. The body column carries the encoded AnyValue payload; body_kind is the cheap routing flag the query planner uses to decide whether reconstruction is defined for this row.

Body representation (AnyValue handling)

OTLP’s LogRecord.body is AnyValue — string, bool, int, double, bytes, array, or kvlist. The spec is explicit (Logs Data Model §Body): “Body MUST support AnyValue to preserve the semantics of structured logs emitted by the applications.” Real OTel emitters send structured Body routinely, not just text.

Ourios distinguishes two body shapes at ingest:

• body_kind = String — LogRecord.body is AnyValue::String. The miner runs the §6.2 algorithm over the unwrapped string: tokenize, mask, descend the tree, attach to or create a leaf. params, separators, confidence, lossy_flag are populated per the existing semantics.
• body_kind = Structured — LogRecord.body is any other AnyValue variant (kvlist, array, int, double, bool, bytes). The miner does not run the §6.2 algorithm. The body is encoded with the Ourios canonical body encoding (see The Ourios canonical body encoding below) and stored in the body column; no template is mined, no params/separators are emitted. template_id is allocated per the Template-key composition rule below — for this branch the key is (severity_number, scope_name, BodyKind::Structured), so all structured-Body records sharing a (severity, scope) share one template_id. The leaf the id points at carries the Structured marker and an empty body_template. confidence = 1.0 (sentinel), lossy_flag = false (the canonically-encoded body is authoritative; nothing is reconstructed from a template).

This is the conservative default. It preserves the structural content of the body (the Ourios canonical body encoding below makes [§3.3] reconstruction well-defined for the structured branch: stored_bytes ↔ AnyValue is bidirectional and byte-deterministic), it avoids inventing template structure for arbitrary AnyValue trees, and it sidesteps the spec ambiguity of “what is the template for {"msg": "x", "user_id": 42}.” A future opt-in mine-inner-field mode (e.g., mine body.kvlist["msg"] as the line if present) is a configurable knob, not the default; that decision lives with the maturity-stage move from red → green once corpus evidence informs which inner-field conventions are worth specifying.

A third path — render-to-string + mine (canonicalise structured Body to JSON-ish text and run it through the §6.2 mining algorithm) — was rejected because mining over the JSON serialisation produces token templates that depend on the serialiser’s whitespace and field-ordering choices, which is both fragile (changing serialisers shifts every template) and defeats the §3.3 reconstruction guarantee for any record where the original wire form was protobuf rather than JSON. Storing the canonical encoding (without mining over it) is different from this rejected path: storage is faithful, it just doesn’t get a template extracted.

The Ourios canonical body encoding (body_kind = Structured).

Amendment 2026-06-09 (no canonical OTLP JSON exists). This paragraph previously called the encoding “the OTLP-canonical JSON encoding per the OTLP specification’s HTTP/JSON binding,” implying a spec-defined canonical form. There is none. Per the OTLP spec, the OTel common docs, and a maintainer answer (Josh Suereth, 2026-06-09): OTLP/JSON is the proto3 JSON mapping plus a short closed list of deviations (hex trace_id/span_id, integer enums, ignore-unknown-fields, lowerCamelCase) — with no normative rules on whitespace, key/field ordering, or number canonicalisation, and OTLP does not require lossless translation between formats. The text below is reframed to state the rule as an Ourios-local deterministic encoding, not an OTLP conformance point, and renamed to “the Ourios canonical body encoding.” No code and no RFC status change here; the encoder is ourios-core’s otlp::canonical (a separate follow-up PR aligns its doc comments).

Amendment 2026-06-11 (doubles round-trip bit-exactly — #130). The rule below previously left double precision implicit, and decode(encode(x)) drifted 1–2 ULP for ~12% of arbitrary finite f64. Investigating #130 located the loss on the decode side, not the encoder: the emitter already produces shortest-round-trip digits (serde_json’s Ryu f64 formatter — with-serde adds no custom double formatter), but serde_json’s default float parsing is approximate. The rule now pins both halves: doubles are emitted as shortest-round-trip JSON numbers and decoded with correctly-rounded float parsing (serde_json’s float_roundtrip feature, declared load-bearing in ourios-core), so the f64 round-trip is exact for every finite double — the faithfulness guarantee below holds for arbitrary doubles, not just “nice” ones. Non-finite doubles (NaN, ±∞) have no JSON-number form and encode as null — bytes that do not decode back; that pre-existing gap is explicitly out of scope here and stays open. No RFC status change.

The body column carries the Ourios canonical body encoding of the AnyValue: a proto3-JSON form, defined below. This is an Ourios-local deterministic convention, not an OTLP-mandated canonical form. OTLP defines no canonical or byte-deterministic JSON encoding. Its only normative JSON rules are the proto3 JSON mapping (per the protobuf spec) plus a short closed list of OTLP-specific deviations — trace_id / span_id as hex strings (not applicable to a body AnyValue, which carries no IDs), enum values as integers, ignore unknown fields, and lowerCamelCase field names. The spec is silent on whitespace, key/field ordering, and number canonicalisation, and OTLP does not require lossless translation between formats. There is therefore no “canonical OTLP JSON” to reference; the byte-stable encoding below is Ourios’s own, chosen so the body column is byte-deterministic (storage dedup) and the [§3.3] reconstruction guarantee is well-defined.

The concrete rule is the proto3 JSON mapping as emitted by opentelemetry-proto’s with-serde feature via serde_json:

• field names in lowerCamelCase;
• int64 / uint64 values as decimal strings (proto3 JSON’s canonical emit form; decoders accept a JSON number or string);
• double values as JSON numbers in shortest-round-trip form, decoded with correctly-rounded float parsing — decode(encode(x)) is bit-exact for every finite f64 (#130; see the 2026-06-11 amendment above);
• bytes as base64;
• KvlistValue and ArrayValue element order preserved as received — not sorted (this is explicitly not RFC 8785 / JCS canonical JSON);
• byte-deterministic because the proto types have a fixed serde field order and serde_json serialisation is deterministic.

Canonical byte examples: {"intValue":"-42"}, {"doubleValue":2.71}, {"boolValue":true}, {"bytesValue":"<base64>"}, {"arrayValue":{"values":[…]}}, {"kvlistValue":{"values":[{"key":"…","value":{…}}]}}.

“Deterministic” here means byte-identical. Re-encoding the same in-memory AnyValue with this encoder yields byte-for-byte identical output (within a fixed opentelemetry-proto / serde_json version). This is the byte-level reading, not a weaker struct-level one: it is what lets the body column be deduplicated and lets two receivers handling the same logical AnyValue produce the same stored bytes. The receiver path: OTLP/gRPC (protobuf wire) decodes to an in-memory AnyValue and re-encodes here; OTLP/HTTP+JSON decodes the incoming JSON to an in-memory AnyValue and re-encodes the same way, so the stored bytes do not depend on the producer’s whitespace, field order, or int64-as-number-vs-string choice.

The faithfulness guarantee — stored_bytes decode back to the original in-memory AnyValue — is an Ourios guarantee delivered by this encoder/decoder pair, not an OTLP lossless promise (OTLP makes none). lossy_flag = false for structured rows rests on this Ourios guarantee, not on any OTLP conformance claim.

Duplicate keys. OTLP KvlistValue is a repeated KeyValue that the data model treats as a map with unique keys; the data-model map equality is order-insensitive, but OTLP does not define wire-order equality, which is why preserving received order (above) is the safe, spec-permitted choice. A KvlistValue carrying duplicate keys is non-conforming OTLP input with no defined semantics. Ourios does not silently dedup or reorder such input: it preserves the entries verbatim in the encoding (so the round-trip stays faithful) and makes no map-semantic guarantee for it, rather than inventing one.

Template-key composition

A template’s identity (the discriminator the Drain tree uses to decide “is this the same template?”) depends on the body shape:

• body_kind = String — key tuple is (severity_number, scope_name, masked_body_tokens).
• body_kind = Structured — key tuple is (severity_number, scope_name, BodyKind::Structured). All structured-Body records sharing a (severity_number, scope_name) share one template_id. This intentionally forfeits structured-body shape clustering — the rationale is that the structured-Body branch’s value comes from the faithful preservation of attributes and the canonically-encoded body, not from grouping similar AnyValue shapes. Operators who need shape-level clustering can opt into a future body_shape_fingerprint column (a stable hash over the AnyValue’s structural skeleton — kvlist key-set, nested shape, leaf-type sequence; values ignored) as a reserved extension; the gate for adding it is “we have a concrete consumer,” not “it might be useful.”

The bullet rationale below applies to both branches:

• severity_number is part of the key because INFO and ERROR versions of the same body text are semantically distinct events. “user logged in” at INFO is a routine signal; “user logged in” at ERROR is an alarm (or an emitter bug) — collapsing them to one template_id would surface either as the other on query, which is a [§3.1] “no silent merges” violation in disguise. The OTLP-spec-valid severity_number = 0 (UNSPECIFIED) is a distinct key value, not coalesced with any specified severity.
• scope_name is part of the key because the same body text emitted from two different instrumentation scopes (myapp.login vs myapp.checkout) describes two different events. The scope is the OTel-canonical “which code path emitted this,” directly analogous to the package/logger name in traditional logging frameworks. Records with no scope (scope_name = None) cluster as their own (severity, None) bucket.
• resource_attributes are NOT part of the key. They identify who sent the record (service, host, k8s pod), not what event was emitted. The tenant_id derivation (below) already encodes the partition decision over Resource. Folding Resource into the template key would explode template cardinality proportionally to the deployment fleet size without adding semantic discrimination — the same myapp.login template from two replicas of service.name = api is the same template.
• event_name is not in the key today but is reserved as a candidate addition. RFC 0001 stays at the OTLP-canonical severity+scope key; promoting event_name into the key is a follow-up RFC patch once corpus evidence justifies it.

The Drain tree’s implementation of this tuple (extra prefix levels above length-N, tuple-keyed leaf lists, separate trees per (severity, scope), etc.) is §6.2 implementation territory and may be revisited based on cardinality observations from the corpus benchmark. The RFC pins only the semantic key.

Tenant derivation

tenant_id is derived per ResourceLogs group, not per OTLP export batch. Each ResourceLogs carries its own Resource.attributes, and a single OTLP export can contain multiple ResourceLogs groups from different sources — so one export can route records to multiple tenants. The derivation runs once per inherited Resource; the resulting tenant_id applies to every LogRecord under that ResourceLogs group (across all its ScopeLogs), and the receiver fans the records out into per-tenant streams.

The default per-Resource rule:

tenant_id := resource.attributes["service.name"]   if present
          ?: <operator-required fallback rule>

service.name is the conventional OTel unit of “what application emitted this,” and it maps directly onto Ourios’s per-tenant template-tree partitioning ([§3.7]). Operators with a different multi-tenant model (per-namespace, per-customer-id-attribute, composite of multiple attributes) configure an alternative rule; the receiver does not invent a tenant identity that the operator hasn’t declared.

If a ResourceLogs group’s Resource resolves to no tenant under either the default rule or the operator’s fallback, the receiver rejects the entire export batch with a controlled error (no panic, no silent assignment to a “default” tenant; the sender sees the failure and either fixes its emitter or its deployment). Per-Resource rejection within an otherwise-valid batch is not supported in this RFC — the all-or-nothing failure mode is simpler to reason about for the sender, and OTLP’s batch-level acknowledgement model fits all-or-nothing more naturally than partial-success. The receiver-side specification of this rejection path (and any future opt-in for partial acceptance) lives in RFC 0003 — OTLP receiver (forthcoming).

Template identity

template_id is a cluster-wide unique monotonic u64 (with each tenant seeing a monotonic subsequence), allocated when a new leaf is created and never reused or reassigned. The id space is shared across tenants so that the same u64 value never refers to two different leaves; the per-tenant subsequence guarantee preserves [§3.7] by making each tenant’s allocation order observable in isolation. Cross-tenant content identity is intentionally not guaranteed — two tenants emitting the structurally identical template (same (severity_number, scope_name, masked_body_tokens) tuple) will have different template_ids, so a template_id alone never links structurally-equivalent templates across tenants. (The u64 value itself is cluster-wide unique, per the previous paragraph; what is not guaranteed is that the same template across two tenants resolves to the same id.) This preserves [§3.7] (per-tenant template trees) by construction; cross-tenant analytics that need content identity (deduplication across tenants for storage savings, shared template dashboards) are an opt-in concern and are not provided by the miner. A future template_fingerprint side column may carry a canonical content hash over (severity_number, scope_name, masked_body_tokens) for opt-in cross-tenant use; the gate for adding it is “we have a concrete consumer,” not “it might be useful.”

Template version. template_version starts at 1 when the template is created and increments by 1 on every widening event: either a new wildcard slot opens (a previously fixed token at position i becomes <*>), or an existing wildcard’s typed parameter set changes (e.g. a <NUM> slot starts seeing <STR> values). To detect the second case, every leaf carries — alongside its template — a slot_types: Vec<HashSet<ParamType>> indexed by wildcard slot, recording every ParamType observed in that slot. A type expansion is the addition of a ParamType to one of these sets. The pair (template_id, template_version) uniquely identifies one structural state of a template. Queries against template_id = X return all versions; queries against (template_id, template_version) = (X, V) return only the named state. The DSL surface is RFC 0002’s concern, not this RFC’s, but the data model must support both.

Why two integers and not a content hash. A content hash makes identity global by construction; in a multi-tenant backend that is a tenant-isolation leak rather than a feature. A content hash also makes template_version redundant — once the canonical template string changes, the hash changes, so versioning collapses into alias-mapping between hashes. Per-tenant monotonic ints with an explicit version field are smaller in the Parquet column, easier to reason about under [§3.7], and keep (template_id, template_version) as a meaningful compound key.

6.2 Algorithm

Amendment 2026-05-13. Rewritten to take a structured OTLP LogRecord rather than a raw &str, in line with the §6.1 amendment. The algorithm now opens with the body.kind fork from §6.1’s Body representation: AnyValue::String runs the Drain mining steps (the prior algorithm, preserved verbatim below); every other AnyValue variant short-circuits to the structured emit per §6.1’s Template-key composition fork. Step 3’s descent now incorporates (severity_number, scope_name) into the tree key, again per §6.1 — the implementation choice (extra prefix layers, tuple-keyed leaf lists, separate trees per (severity, scope)) stays in §6.2 as the algorithm’s responsibility, but the semantic key is pinned by §6.1. The ingest signature on MinerCluster becomes ingest(record: &OtlpLogRecord); pre-amendment callers were ingest(tenant_id, raw: &str).

The miner sees an already-tenant-resolved (tenant_id, record: OtlpLogRecord) pair. The receiver (RFC 0003) is responsible for resolving tenant_id per ResourceLogs and fanning records into per-tenant streams before the miner sees them; §6.1’s Tenant derivation pins that contract.

For each ingested OTLP LogRecord:

0.  match record.body.kind:

      AnyValue::String(s):
          # Continue with the Drain mining algorithm in steps
          # 1–5 below, treating `s` as the `L_raw` of the prior
          # spec. body_kind = String.

      AnyValue::Bool | Int | Double | Bytes | Array | KVList:
          # Structured short-circuit per §6.1 *Body
          # representation*. The miner does NOT run the Drain
          # mining steps. body_kind = Structured.
          encoded = encode_canonical_body(record.body)
              # Ourios canonical body encoding (a proto3-JSON form;
              # lowerCamelCase fields, int64/uint64 as decimal
              # strings, bytes as base64, kvlist/array order
              # preserved — NOT sorted). This is an Ourios-local
              # deterministic convention, NOT an OTLP-mandated
              # canonical form: OTLP defines no canonical JSON and
              # requires no lossless translation. For records over
              # OTLP/gRPC the receiver decodes protobuf and
              # re-encodes here; for OTLP/HTTP+JSON it decodes to
              # the in-memory AnyValue and re-encodes the same way,
              # so stored bytes are byte-identical regardless of the
              # producer's whitespace / field order / int64 form.
              # The lossy_flag = false promise rests on this Ourios
              # round-trip guarantee — see §6.1 for the why.
          template_id = allocate_or_reuse_structured_template_id(
              record.severity_number,
              record.scope_name,
          )
              # Per §6.1 *Template-key composition*, the
              # structured-Body key is (severity_number,
              # scope_name, BodyKind::Structured). All structured
              # records sharing a (severity, scope) share one
              # template_id. The leaf the id points at carries the
              # `Structured` marker and an empty body_template.
          attach_structured(record, encoded, template_id,
                            confidence = 1.0,
                            lossy_flag = false)
              # confidence = 1.0 sentinel; lossy_flag = false
              # because the canonicalised body is authoritative,
              # nothing is reconstructed from a template.
          return

1.  L_tok, separators = tokenize(L_raw)
        # tokenize splits on Unicode whitespace only — every
        # codepoint matching `char::is_whitespace()` (ASCII space,
        # tab, CR, LF, plus the broader Unicode whitespace classes
        # U+0085, U+00A0, U+1680, U+2000–U+200A, U+2028, U+2029,
        # U+202F, U+205F, U+3000). Every other byte (including
        # punctuation such as `=`, `:`, `,`, `;`, `[`, `]`, `(`,
        # `)`) stays inside a token; structured separators are the
        # masking layer's responsibility (§4.2 / step 2). The
        # captured whitespace runs go into `separators` so that
        # reconstruction (§6.6) is byte-identical.
        # On failure (malformed UTF-8, embedded NUL, line longer
        # than max-line-bytes): emit a parse-failure record and
        # increment ourios.miner.parse_failures. Skip the rest.
        # Note: an empty-after-whitespace string (the AnyValue
        # carries `""` or only whitespace) is not a parse failure
        # — it has zero tokens and the miner short-circuits with
        # the cluster's `NO_TEMPLATE` sentinel rather than
        # descending the tree. The pre-amendment cluster code
        # already routes this case; the spec just records it.

2.  L_masked, typed_params = mask(L_tok)
        # mask applies the configured masking rules in order;
        # any token matching a rule is replaced with its type
        # tag (e.g. <IP>) and the original bytes are pushed
        # into typed_params with that tag. Unmasked tokens
        # remain literal.

3.  parent = tree.descend(record.severity_number,
                           record.scope_name,
                           len(L_masked),
                           L_masked[0..d-1])
        # Per §6.1 *Template-key composition*, the discriminator
        # for "is this the same template?" is the tuple
        # (severity_number, scope_name, masked_body_tokens).
        # Step 3 incorporates severity_number and scope_name into
        # the descent key alongside the masked-token prefix used
        # by published Drain. The implementation may layer extra
        # prefix levels above the length-N node, key leaf lists
        # by (severity, scope), or maintain separate trees per
        # (severity, scope) — the choice is cardinality-driven
        # and revisitable from corpus observations. The
        # severity_number = 0 (UNSPECIFIED) and scope_name = None
        # cases are valid distinct key positions; they cluster as
        # their own buckets, never coalesced with any specified
        # severity or named scope.
        # if a node along the path does not exist, create it.

4.  candidate = argmax over leaf in parent.leaves of
                  simSeq(L_masked, leaf.template)
    if candidate is None:
        # no leaves under parent yet; create one. Creation does not
        # emit an audit event — `ourios.miner.template.count` already reflects
        # leaf allocation, and §6.4 reserves the audit stream for
        # widening events whose semantics need cross-referencing.
        leaf = new Leaf(template = L_masked)
        parent.leaves.push(leaf)
        # On fresh-leaf creation the template is L_masked verbatim,
        # so every <*> in it came from mask(); params == typed_params.
        attach(L_masked, typed_params, separators, leaf,
               confidence = 1.0, lossy_flag = false)
        return

5.  similarity = simSeq(L_masked, candidate.template)
    confidence = similarity / threshold

    if similarity >= threshold:
        # clean or lossy attach; widen the template if needed.
        # widen() returns:
        #   widened           — the new template (existing fixed
        #                       positions that mismatched L_masked
        #                       become <*>)
        #   new_wildcards     — the set of positions that just
        #                       became <*> (the audit payload)
        widened, new_wildcards = widen(candidate.template, L_masked)
        if new_wildcards > 0:
            candidate.template = widened
            candidate.version += 1
            emit_audit(event_type = template_widened,
                       template_id = candidate.id,
                       old_version, new_version = candidate.version,
                       positions_widened = new_wildcards.positions,
                       ...)
            ourios.miner.merges.inc()

        # Build the params array. One entry per <*> in the (possibly
        # just-widened) template, in template order. For each slot:
        #   - if the slot existed before this attach AND mask() emitted
        #     a typed_params entry for it, use that entry verbatim.
        #   - if the slot is a fresh wildcard from this widening (the
        #     position held a literal token in candidate.template before
        #     the widen call), the original literal at that position in
        #     L_tok is captured as { type_tag: STR, value: L_tok[pos] }.
        # Without this step §6.1's "one entry per <*> slot" invariant
        # is violated and §6.6's reconstruct() has no value to insert
        # at the freshly-widened position.
        params = build_params(candidate.template, typed_params,
                              L_tok, new_wildcards)

        # Type-expansion: if any wildcard slot now sees a typed param
        # whose type tag is not already in that slot's observed-type
        # set, widen the slot's type set, bump the version, and
        # audit. The leaf carries `slot_types: Vec<HashSet<ParamType>>`
        # alongside its template (data model in §6.1).
        new_types = update_slot_types(candidate, typed_params)
        if not new_types.is_empty():
            candidate.version += 1
            emit_audit(event_type = template_type_expanded,
                       template_id = candidate.id,
                       old_version, new_version = candidate.version,
                       slots_expanded = new_types,
                       ...)

        attach(L_masked, params, separators, candidate,
               confidence,
               lossy_flag = false)  # §6.6: lossy_flag is set only on
                                    # tokenizer/preprocessing failure,
                                    # not on confidence < 1.0
        return

    if similarity >= floor:
        # lossy zone: the line is "close" but doesn't meet
        # threshold. Create a new leaf rather than force-merging.
        # Body retention is unconditional in this branch.
        leaf = new Leaf(template = L_masked)
        parent.leaves.push(leaf)
        # As in the candidate-is-None branch, the new leaf's template
        # is L_masked verbatim, so params == typed_params.
        attach(L_masked, typed_params, separators, leaf,
               confidence,
               lossy_flag = false,
               body = L_raw)  # forced body retention
        ourios.miner.body_retention.utilization.observe(retained = true)
        return

    # similarity < floor: parse failure
    ourios.miner.parse_failures.inc()
    emit_failure_record(L_raw, reason = "no candidate above floor")

Branching invariants:

• Step 0’s structured short-circuit never enters the Drain mining steps (1–5). Structured-Body records do not widen, do not emit template_widened or template_type_expanded audit events, do not contribute to ourios.miner.merges, and never carry params/separators. The structured branch’s [§3.1] preservation is vacuous: no template merge happens, so no silent merge is possible.
• The tree only deepens on first observation of a (severity_number, scope_name, length, prefix tokens) shape (the §6.1 template-key tuple, anchored at this section’s step 3).
• Leaves are split (new leaf created) when the best candidate is in the lossy zone; they are never split when the candidate is clean.
• A leaf is widened (wildcards introduced) when a clean attach would otherwise mismatch positions. Every widening emits an audit event (§6.4).
• A leaf’s wildcard slot is type-expanded when an attach maps a typed parameter whose ParamType is not already in that slot’s slot_types[slot] set. Type expansion increments template_version and emits a template_type_expanded audit event (§6.4).
• A single attach can trigger both wildcard-widening and type-expansion in the same leaf; in that case template_version increments twice and two audit events are emitted, in that order.
• The leaf’s template_version only increments on widening or type-expansion, not on a clean attach. Structured-Body leaves are never widened or type-expanded; their template_version stays at 1 for the lifetime of the leaf.

6.3 Confidence scoring [§3.1]

confidence = simSeq / threshold. The ratio framing makes the decision boundary land at confidence == 1.0 regardless of the configured threshold, which gives ourios.miner.confidence.p50 and ourios.miner.confidence.p01 ([§3.1] required metrics) a stable interpretation across tenants with different thresholds: the p01 value tells you how close the bottom 1% of attaches are to the merge boundary. A collapsing p01 means many lines are barely passing — a tuning signal even though the threshold itself has not moved.

Three zones, with concrete defaults:

• confidence ≥ 1.0 (i.e. simSeq ≥ threshold): clean attach. No body retention.
• floor / threshold ≤ confidence < 1.0: lossy zone. The line attaches to a freshly created leaf rather than being force-merged into the candidate (see §6.2 step 5). body is retained unconditionally; lossy_flag follows the §6.6 rule (set only on reconstruction failure, not on lossy zone alone — the body is available either way).
• confidence < floor / threshold: parse failure. ourios.miner.parse_failures increments; the line is written to a failure record with the original bytes intact.

Defaults. threshold = 0.7, floor = 0.4. The threshold floor is fixed by [§3.1] (“threshold ≥ 0.7, lowering requires an RFC, not a config change”); the lossy-zone floor is a tuning knob between threshold and 0. floor = 0.4 matches the paper’s reported default threshold, on the reasoning that lines below the paper’s own bar are likely genuinely different events. Tuning the floor is a per-tenant config decision; it is not load-bearing for any invariant.

6.4 Merge policy [§3.1]

A template widening (per §6.2 step 5) is the operation that [§3.1] calls a “merge.” Every widening emits an audit event with the schema:

{
  event_type: AuditEventType,  # enum:
                               #   template_widened
                               #   template_type_expanded
                               #   template_widening_rejected_degenerate
  tenant_id: TenantId,
  template_id: u64,
  old_version: u32,
  new_version: u32,
  old_template: String,        # canonical form, with <*> for wildcards
  new_template: String,
  triggering_line_hash: [u8; 16],  # blake3 of L_raw, for cross-ref
  triggering_line_sample: Option<String>,  # first 256 B of L_raw
  positions_widened: Vec<u16>, # token positions that became <*>
                               # (empty for template_type_expanded)
  slots_expanded: Vec<SlotExpansion>,
                               # slot index + newly added ParamType(s)
                               # (empty for template_widened)
  timestamp: SystemTime,
}

event_type is the field §6.7’s drift-detection query filters on. ourios.miner.merges increments on every event whose event_type is template_widened or template_type_expanded (the two structural widenings); template_widening_rejected_degenerate is recorded but does not increment ourios.miner.merges. The audit stream is written to the same WAL as the data records and ends up in a dedicated audit Parquet file per tenant per compaction window (schema in ourios-parquet’s RFC, not this one).

Default policy: strict. Widening is permitted whenever the clean-attach path in §6.2 would otherwise mismatch positions. The audit event is mandatory — no widening, of any reason, ever proceeds without one. Code paths that would emit a widening without emitting an audit event are blocked at PR review per hazards.md H1.

WAL durability ordering of audit events. A single attach may emit two audit events in order (RFC0001.7: template_widened immediately followed by template_type_expanded) and one data record. The contract this RFC requires from the future ourios-wal RFC is an ordering-plus-durability-barrier: a data record carrying template_version = V must not become durable before every audit event justifying the leaf’s progression to V is durable. Crash recovery may then observe some prefix of [event_1, event_2, …, data_record], but never a state in which the data record exists without the events that caused its version stamp. Any framing strategy that satisfies this — a composite multi-record frame, batched-fsync ordering, a two-phase write-then-link, anything else — is acceptable; the framing is ourios-wal’s choice, the ordering barrier is this RFC’s requirement. Without it, replay would bump template_version fewer times than the in-memory leaf did and the surviving data records would reference a version the audit stream cannot substantiate.

Degenerate template guard. If a widening would leave the template with zero non-wildcard tokens (the entire template becomes <*> <*> … <*>), the widening is rejected, the line is treated as a parse failure (confidence = 0, retain body, increment ourios.miner.parse_failures), and an audit event with event_type = template_widening_rejected_degenerate records the rejection. A fully-wildcard template provides no parsing value and would swallow arbitrary lines.

6.5 Parameter handling [§3.2]

Per-parameter byte limit. Default 256 B, configurable up to 1 KiB (the [§3.2] ceiling). Above 1 KiB requires an RFC.

Overflow behaviour. When a parameter value (post-masking) exceeds the configured limit, the parameter slot is replaced by an OVERFLOW marker:

Param {
  type_tag: ParamType::OVERFLOW,
  value: encode(length: u32, sha256_prefix: [u8; 8]),
}

The original line L_raw is captured into the body column unconditionally (overflow forces body retention, regardless of lossy_flag). The 8-byte SHA-256 prefix lets queries “find rows where this exact long parameter occurred” without storing the long value in the columnar data. Reconstruction honours overflow: reconstruct(record) falls back to body when any param has type_tag == OVERFLOW.

Telemetry. Two metrics for [§3.2] and hazard H2:

• ourios.miner.params.overflow (counter, attributes ourios.tenant, ourios.service): increments per overflow.
• ourios.miner.params.overflow.utilization (gauge, attributes ourios.tenant, ourios.service): rolling overflow rate. Alert at > 0.01 per service per [§3.2].

6.6 Body reconstruction [§3.3]

Amendment 2026-06-08 (reader render contract). H7.3 (§5) referenced “the §6.6 warning marker,” but §6.6 defined no such marker — the prose Reader behaviour paragraph it replaced described one only informally. This amendment adds the Reader render contract subsection below, defining the marker as a structured, out-of-band per-row Reconstruction signal (Faithful | RetainedVerbatim) the reader attaches to the rendered row — never a mutation of the body bytes — and pinning the lossy short-circuit H7.3 requires (return body verbatim, do not call reconstruct). The clean-path read-time template lookup (a registry mapping (template_id, template_version) → tokens at read time) is explicitly out of scope and deferred to the querier’s reader-materialisation story (RFC 0007). RFC 0001 stays specified; this clarifies the §6.6 contract, it does not change the on-disk schema or the mining algorithm.

Capture, always. Every successful tokenization in §6.2 step 1 populates the separators array with the bytes between adjacent tokens (and the leading and trailing bytes of the line). The array length is tokens.len() + 1. There is no whitespace heuristic and no “is this whitespace trivial” decision — the bytes are captured verbatim. Storage cost is bounded (typical separator is one space; the array dictionary-encodes well in Parquet) and the implementation has no fuzzy boundary that could decide to drop bytes silently.

Reconstruction function.

fn reconstruct(record: &Record) -> Bytes {
    if record.lossy_flag {
        return record.body.expect("lossy implies retained body");
    }
    if record.params.iter().any(|p| p.type_tag == OVERFLOW) {
        return record.body.expect("overflow implies retained body");
    }
    let template = lookup(record.template_id, record.template_version);
    let mut out = BytesMut::new();
    out.extend_from_slice(&record.separators[0]);
    for (i, token) in template.tokens.iter().enumerate() {
        match token {
            Token::Fixed(s) => out.extend_from_slice(s),
            Token::Wildcard(slot) => {
                out.extend_from_slice(&record.params[slot].value)
            }
        }
        out.extend_from_slice(&record.separators[i + 1]);
    }
    out.freeze()
}

lossy_flag semantics. Set to true if and only if reconstruction is not guaranteed to equal the ingested bytes:

• The tokenizer failed (malformed UTF-8 inside a token, embedded NUL, line exceeded the configured max_line_bytes cap before tokenization completed).
• A preprocessing rule explicitly rejected the line.

The lossy zone in §6.3 (low confidence) does not automatically set lossy_flag: the body is retained either way, and reconstruction from template + params + separators is still expected to match. The flag is reserved for the cases where the record genuinely cannot be reconstructed.

Reader render contract

The functions above run at write time: reconstruct is the property-test oracle and the in-process renderer the miner exercises while the template is in hand. The reader — the read-side path that materialises a stored Parquet row back into the effective original line for a query result — is a distinct caller, and H7.3 pins the contract it must honour. The contract below covers String-body rows (body_kind = String); structured bodies are out of scope (see the end of this subsection).

For a String-body row, the render result is (bytes, reconstruction). Rendering yields two things: the effective original line bytes and a per-row reconstruction signal

#![allow(unused)]
fn main() {
enum Reconstruction {
    Faithful,         // bytes == ingested line, reconstructed from template
    RetainedVerbatim, // bytes are the retained `body` column, not reconstructed
}
}

The Reconstruction signal is the “§6.6 warning marker” that H7.3 references. It is structured, out-of-band metadata attached to the rendered row — not a mutation of the body bytes. A consumer (the DSL output layer, RFC 0007; a UI) renders RetainedVerbatim as a “rendered from the retained body bytes, not reconstructed” warning beside the row, exactly as it would render any other per-row annotation.

A body-byte annotation is explicitly rejected as the marker. Prefixing the body with a sentinel string, wrapping it in a marker character, or otherwise editing the bytes to carry the warning would break the verbatim guarantee: the whole point of the lossy path is that an operator asking “show me what was actually logged” gets the ingested bytes back unchanged [§3.3]. The marker therefore lives beside the bytes, never inside them.

Lossy path (H7.3). For a row with lossy_flag = true — the tokenizer-failure / explicit-rejection cases enumerated under lossy_flag semantics above — the reader returns the body column verbatim with Reconstruction::RetainedVerbatim, and does not invoke reconstruct: no template lookup, no token walk. The same short-circuit applies to a String-body row carrying an OVERFLOW param (§6.5): its retained body is returned verbatim with Reconstruction::RetainedVerbatim. reconstruct’s own lossy_flag / OVERFLOW early returns remain in place as a defensive guard for callers that reach it anyway; the reader’s contract is to short-circuit before that call, so the guard is belt-and-braces, not the primary mechanism.

Structured bodies are out of scope of this amendment. A body_kind = Structured row renders its body column from the Ourios canonical body encoding (the §6.1 rule, exercised by the RFC0001.9 structured short-circuit). This amendment defines the Reconstruction marker for the implemented String path only; how a structured render maps to a Reconstruction signal is left open here, to be settled when structured-body rendering is wired.

Clean path. For a faithful row (lossy_flag = false, no OVERFLOW param, template available) the reader invokes reconstruct(record) and attaches Reconstruction::Faithful. This path requires resolving (template_id, template_version) → tokens against a template registry available at read time. That registry — how the reader obtains the template a stored row was mined against, given that the miner’s in-memory tree is an ingest-side structure — is a separate concern and out of scope of this amendment. Today reconstruct is exercised only where the template is already in hand (the write-side property test H7.1, and H7.4); the read-time lookup mechanism is future work, tracked with the querier’s reader-materialisation story (RFC 0007). This amendment pins the render contract and the lossy path; it does not specify a template-registry design.

The reader never silently substitutes one rendering for the other: every rendered String-body row carries its Reconstruction signal, and the lossy/clean branch is selected solely by the row’s lossy_flag (and the §6.5 OVERFLOW case), never inferred from the bytes.

Property test. For every row r in the corpus where r.lossy_flag == false:

reconstruct(r) == ingested_bytes(r)

Failure is a build break, not a regression — [§3.3] and hazard H7 both name this as the property test that gates merges.

6.7 Template versioning and drift [§3.5], hazard H5

A template’s structural state changes over time as widenings (§6.4) and parameter-type expansions accrue. Each change increments template_version and emits an audit event of type template_widened or template_type_expanded. ourios.miner.template.version_changes counts these.

Two distinct cross-cutting questions. “Same leaf, different structural snapshots” and “different leaves that mean the same thing” are separate problems, with separate query forms:

• Cross-version (within one leaf). A leaf’s template_id is stable across every widening of that leaf (§6.1, “Template identity”); only template_version advances. So a literal predicate where template_id = X already returns rows from every version of leaf X by construction — no alias resolution required. To pin to a single structural snapshot, query where (template_id, template_version) = (X, V).
• Cross-alias (across leaves). When a deploy changes a log line enough that the miner allocates a new leaf instead of widening the existing one, the operator has two template_ids for what is semantically the same template. Resolving “all rows for the thing X represents” then requires walking an alias set that spans leaves. RFC 0002 §5.4 exposes this as where template_id.resolves_to(X); bare where template_id = X does not follow alias chains.

The data model in §6.1 supports both shapes: cross-version is free because template_id is stable across widenings; cross-alias is served by a separate alias index that maps a representative template_id to the equivalence class of template_ids the operator (or a future inference layer) considers semantically the same template.

Alias index lifecycle. Cross-alias is structurally distinct from cross-version: widening is intra-leaf and increments template_version (one template_id, several versions); aliasing is inter-leaf and groups template_ids the miner allocated separately (different template_ids, the operator asserts they mean the same thing). template_widened events therefore do not populate the alias index — they live on the cross-version axis.

Alias write path: operator-driven and audited [§3.1]. The alias index is operator-driven, never silently inferred. An alias assertion — “leaf B is the same template as leaf A” — is an explicit operator action recorded as an audited, durable event, exactly as §3.1 requires of any merge (“every merge emits an audit event; explicit”). Automatic inference is the precise failure mode §3.1 forbids: an auto-aliased cross-semantic merge is a silent merge by another name. The amendment below replaces the previous “no creation event / produced out of band” deferral with the mechanism.

Amendment (alias write path, 2026-06-07). Resolves the §9 open question “Alias index creation mechanism.” The alias index is produced by operator-driven, audited, reversible assertions on the §6.4 audit stream; automatic inference is deferred to a possible future propose → operator-confirm layer (proposals never enter the active index unconfirmed). Adds the alias_asserted / alias_retracted audit events, the per-tenant alias-map projection the querier reads, and §5 scenarios RFC0001.12–RFC0001.16. The RFC re-enters the ladder at specified until those scenarios land (precedent: the RFC 0003 served-binary amendment).

The alias model. An alias set is an equivalence class of template_ids within one tenant [§3.7]. Membership — which template_ids are in the class — is the only thing that carries contract weight: resolves_to expands by membership and nothing else (RFC0001.13). The class also has a canonical representative, defined as the numerically smallest member of the materialized set. The canonical representative is a derived display/identity convenience, not what defines membership: it gives the set a stable name and is deterministic regardless of assertion order, but it plays no part in deciding who belongs to the class. This derivation rule is an evolvable implementation detail, not a contract — it may change (for example, letting an operator designate a preferred representative) without changing alias-set semantics, precisely because resolves_to expands by membership regardless of which member is canonical. (This is distinct from the event-level representative_id below, which is merely the operator’s anchor id for an assertion and need not equal the derived canonical.) Membership is cross-leaf only: it groups template_ids the miner allocated as separate leaves. It never crosses the cross-version axis — every version of a single leaf already shares one template_id (§6.1, “Template identity”), so template_version is not an alias concern and the alias index holds no template_version field. The two axes stay disjoint: widenings move a leaf along template_version; aliasing groups distinct template_ids.

The assertion event. Aliasing is expressed by two new AuditEventType variants on the §6.4 audit stream, carrying an alias-specific payload:

{
  event_type: AuditEventType,  # alias_asserted | alias_retracted
  tenant_id: TenantId,
  representative_id: u64,       # operator's anchor id for this
                               # assertion — names one member of the
                               # asserted set; carries no contract
                               # weight beyond that and need not equal
                               # the set's derived canonical (smallest)
  member_ids: Vec<u64>,        # the other ids grouped by / removed
                               # from the set in this assertion
  actor: ActorId,              # operator / API principal that
                               # issued the assertion — aliasing is
                               # never anonymous (§3.1 "explicit")
  reason: Option<String>,      # operator-supplied justification
                               # (e.g. "deploy 2026-06 re-split the
                               # login template"), <= 256 B
  timestamp: SystemTime,
}

The asserted set of an event is the full union {representative_id} ∪ member_ids — representative_id is one named member of that set, never excluded from it. An alias_asserted event groups its entire asserted set into one equivalence class; an alias_retracted event removes every id in its asserted set from their class. Membership is therefore defined purely by the union of ids in the event, independent of which id the operator chose as the anchor.

These events flow through the same audit stream as template_widened (§6.4) and inherit its durability contract: they are written to the WAL and become durable under the §3.4 WAL-before-ack barrier before the assertion is acknowledged to the operator. An assertion that has not hit the WAL is not acknowledged — there are no in-memory-only aliases. Unlike widenings, alias events are not emitted on the ingest hot path; they originate from an explicit operator/control-plane call, so they do not gate attach latency. They do not increment ourios.miner.merges (that counter is reserved for the two structural widenings, §6.4); alias activity is counted separately as the ourios.miner.alias.assertions / ourios.miner.alias.retractions counters enumerated in §6.8’s telemetry table (mandatory, ourios.tenant attribute), keeping operator-driven aliasing first-class telemetry per [§3.1] / §6.3 alongside ourios.miner.merges.

Materialization and storage. The durable alias event log (the alias_asserted / alias_retracted stream above) is the source of truth; the queryable per-tenant alias map is a projection built by folding that log per tenant. Each alias_asserted unions its full asserted set ({representative_id} ∪ member_ids) into one equivalence class, merging any pre-existing classes that share a member so that overlapping assertions converge on a single class regardless of arrival order; each alias_retracted removes its asserted set’s ids from their class. The canonical representative of each materialized class is then derived as min(members) — so it re-derives automatically when membership changes and never depends on which id any event named as its anchor. A class that drops to a single member is no longer an alias set (that lone id resolves only to itself, RFC0001.16). The folded result is persisted as a per-tenant artifact (one map per tenant, not per partition) that the querier reads at compile time — note: that persisted artifact is the deferred cache of the 2026-06-12 amendment below; in v1 the querier folds the map directly from the audit stream and no artifact exists yet. Three venues were considered:

• Per-tenant projection from the audit/alias event log (chosen). Aliases are a tenant-scoped projection rebuilt from the durable event log and persisted as a small per-tenant alias-map artifact under the tenant root. This reuses the §6.4 audit infrastructure end-to-end (no new write plane), matches the tenant scope of the data (§3.7), and keeps the truth (the event log) append-only and replayable while the map is a cache that can always be rebuilt by re-folding the log.
• Tenant-root aliases.parquet / JSON written directly (rejected as the source of truth). A directly-mutated file has no audit trail of its own; it would either duplicate the event log or become an unaudited mutation point, violating §3.1’s “every merge is audited.” It survives only as the serialization format of the projection above — a storage-layer detail (RFC 0005), not the model.
• Extending the partition Manifest (rejected). The manifest is partition-scoped (Manifest { generation, files }, one per partition); alias sets are tenant-global. Putting a tenant-global structure in a partition-local manifest would fragment one logical map across every partition and force N-partition fan-in on every query. Poor fit.

Eventual consistency / read staleness. The querier and the ingester/control-plane run in separate processes (§6.6 names the same seam for the live template registry). The alias map the querier reads is therefore eventually consistent with the latest assertion: an alias asserted or retracted at t becomes visible to queries once the projection is rebuilt and republished, not instantaneously. This is the same staleness window the §6.6 querier↔registry seam already accepts. Staleness is bounded in both directions by the projection refresh / snapshot cadence:

• A not-yet-visible assertion makes resolves_to(X) return a subset of the eventual membership — it never returns rows from a set the operator did not yet assert, so it can only temporarily under-include.
• A not-yet-visible retraction leaves a stale projection still expanding to the old, larger set, so resolves_to(X) can temporarily over-include a member the operator already removed.

Both are transient and self-correcting on the next projection rebuild, and neither cross-contaminates across tenants (§3.7) or fabricates a grouping no operator ever asserted — the over-inclusion is always a previously asserted membership, never a phantom one. The bound on the window (snapshot cadence) is a storage/serving-layer knob, deferred to the RFC 0005 storage decision (see §9).

Amendment 2026-06-12 — the storage decision is made (v1), by the RFC 0005 line. alias_asserted / alias_retracted persist as event_kind 4 / 5 on the RFC 0005 §3.7 audit stream, and in v1 the querier derives the per-tenant alias map at query-compile time by scanning the tenant’s audit stream for those kinds and folding them through this section’s projection semantics (RFC 0005 §3.7.1) — there is no persisted map artifact yet, so “the projection is rebuilt and republished” above reads, in v1, “the audit events are durably written and flushed”; the staleness bound is audit-flush visibility rather than a snapshot/rebuild cadence, with the same bounded under-/over-inclusion directions. The cached per-tenant artifact (file/format/cadence) stays deferred behind the RFC 0009 §3.4 manifest fork; because the event log remains the source of truth, adding the cache later changes no query-visible semantics — the same v1-full-replay-now / accelerate-later shape as §6.9’s snapshot.

Reader / query contract. RFC 0002’s template_id.resolves_to(X) (RFC0002.9, §5.4) loads the requesting tenant’s alias map at compile time and expands X to its alias-set membership: template_id IN {X} ∪ members(set containing X). With no assertions for the tenant, or for an X in no set, the membership is exactly {X} — identical to today’s base-member stub and to bare template_id = X for that id (RFC0001.6 still holds: bare equality never follows alias chains). Expansion is by the set, not by direction: passing any member of a set (representative or not) resolves to the whole set.

Reversibility. An operator can retract an alias; the retraction is itself an alias_retracted audit event with the same durability and isolation guarantees, and the projection drops the retracted membership on its next rebuild. Aliasing and un-aliasing are both explicit, both audited, never silent — closing the loop with §3.1.

sequenceDiagram
    participant Op as Operator / control plane
    participant WAL as WAL (§3.4 barrier)
    participant Log as Alias event log (§6.4 stream)
    participant Proj as Per-tenant alias-map projection
    participant Q as Querier (resolves_to)
    Op->>WAL: assert alias {rep, members, actor, reason}
    WAL-->>Op: ack (only after fsync — §3.4)
    WAL->>Log: append alias_asserted (durable)
    Log->>Proj: fold per tenant (rebuild + publish)
    Q->>Proj: load tenant alias map at compile
    Q->>Q: resolves_to(X) -> template_id IN members(set with X)
    Note over Op,Proj: retraction is the symmetric alias_retracted event

Future work — automatic inference. A later layer may propose aliases from a post-deploy heuristic (e.g. a burst of template_widened / fresh-leaf allocations correlated with a deploy timestamp). Such proposals are never written to the active alias index directly; they enter a review queue and become real only via the same operator-confirmed alias_asserted event specified above. This RFC does not specify that layer — see §9.

Drift detection as a first-class query. “Templates that gained a new version in the window [t1, t2]” is a query against the audit event stream:

SELECT template_id, MIN(old_version), MAX(new_version),
       COUNT(*) AS widening_count,
       MIN(timestamp), MAX(timestamp)
FROM template_audit
WHERE event_type IN ('template_widened', 'template_type_expanded')
  AND timestamp BETWEEN $t1 AND $t2
GROUP BY template_id
ORDER BY widening_count DESC

(SQL shown for spec clarity; the user-visible form is the RFC 0002 DSL, not raw SQL — see hazard H6.) Operators use this query after deploys to spot templates whose structure changed; a sudden cluster of template_widened events correlated with a deploy timestamp is exactly the H5 detection signal.

6.8 Telemetry [§3.1], §6.3

Amendment 2026-06-03. Telemetry export is realigned from a Prometheus client/scrape model to the OpenTelemetry SDK (the maintainer direction recorded against RFC 0009 §3.6 and the roadmap §5 note). This amendment fixes the export architecture and the Prometheus-era terminology (registry → meter provider, scrape → OTLP push, labels → attributes) throughout §§6.8–6.9 and the §5 scenarios.

Amendment 2026-06-08 (dotted-semconv migration — landed). The metric and attribute names are now the dotted-ourios.miner.* scheme, defined in the semconv/registry/ weaver registry alongside the compaction set (RFC 0009 §3.6) and consumed through the generated ourios-semconv constants. The table below lists the registry names. Instrument kinds are unchanged from the original set; the confidence.p50 / confidence.p01 gauges remain in-process views of the confidence histogram (RFC0001.8) — the question of whether they become collector-/backend-derived quantiles over the exported histogram is a genuine contract change to the §3.1.2 mandatory set and stays deferred to its own review, not folded into this rename.

The registry names were audited against the OpenTelemetry semantic-conventions naming rules (the canonical docs/general/{naming,metrics}.md plus the weaver registry policies): counters drop the Prometheus _total suffix and take singular {annotation} units; the two fraction-of-total gauges use the conventional .utilization segment (unit 1) rather than a bespoke .ratio; and the per-line elapsed-time histogram is ourios.miner.duration (UCUM s), the conventional segment for a discrete operation’s elapsed time, not latency.

Export architecture (OTel SDK + OTLP)

Metrics are instrumented through the OpenTelemetry meter API and exported via the OTel SDK’s OTLP metric exporter (push, over OTLP to a collector / endpoint). There is no prometheus client crate and no /metrics scrape endpoint; any Prometheus compatibility is a downstream collector concern, not Ourios’s.

The dependency split follows the standard OTel layering so the heavy SDK and transport crates do not leak into every library:

• Instrumented crates (ourios-miner, ourios-parquet, ourios-ingester, ourios-querier) depend only on the lightweight opentelemetry API crate and resolve instruments through global::meter("ourios.<subsystem>"). No SDK, no OTLP, no transport dependency in a library crate.
• A new ourios-telemetry crate owns the heavy deps — the opentelemetry_sdk and opentelemetry-otlp crates (the upstream package names, underscore and hyphen respectively) plus the OTLP transport. It exposes an init() that builds the OTLP push MeterProvider (periodic-reader export, interval configurable), installs it as the process-global provider, and returns a guard whose shutdown() flushes pending metrics on exit. The binary (ourios-server) calls init() once at start-up; benches and integration tests call the same entry point or substitute an in-memory reader. Adding this crate extends the CLAUDE.md §7 target layout; the new-crate commitment is blessed here, in this RFC, per §7’s rule.

Dimensions are OTel attributes, not Prometheus labels, and OTel splits them in two: resource attributes identify the telemetry producer and are set once on the MeterProvider; data-point attributes vary per measurement. Ourios’s own identity — service.name = ourios-<role> (e.g. ourios-ingester, ourios-querier, matching the role the ourios-telemetry crate initialises the provider for; with service.version, etc.) — is a resource attribute: per the semantic conventions it MUST be set once on the provider’s Resource and MUST NOT be repeated on individual data points.

The per-measurement dimensions in the table below — among them ourios.tenant, the originating service of the ingested logs, and per-metric dimensions like event_type — are data-point attributes. A single ingester multiplexes many tenants and many source services, and [§3.1] / [§3.2] require per-(tenant, service) breakdowns — notably the §6.5 / H2.2 per-service overflow alert — which a single producer-level resource attribute could not provide. The service dimension here is the log’s source service (the value §6.1’s tenant derivation reads), distinct from Ourios’s own service.name — it must not reuse that reserved resource key. It is exported under the dedicated ourios.service attribute key; the tenant dimension is ourios.tenant and the merges change-kind is ourios.miner.template_change (registry enum widened / type_expanded). All three are defined in semconv/registry/attributes.yaml and consumed through the generated ourios-semconv constants.

The mandatory set is defined by the semconv registry: the ourios.miner.* entries in semconv/registry/, surfaced as the generated ourios_semconv::OURIOS_MINER_* constants, are the source of truth for which metrics the miner must expose. Each is registered as an instrument on the ourios.miner meter when the miner is constructed.

OTel’s metric model is collect-on-read: a reader / exporter sees the data points produced during a collection cycle, and an instrument contributes a data point on its first real measurement. The miner emits no synthetic zero-traffic points — so every exported series carries the registry’s required attributes, with no sentinel attribute value to collide with a real tenant / service or violate the template_change enum. §3.1.2 is verified by exercising every instrument with a small representative workload and collecting the metric stream (an SDK in-memory reader in tests): the registry pins the mandatory set; the collection proves each instrument is registered and emits real data under the required attributes.

The metrics enumerated in [§3.1] are mandatory. Full set (the dotted-ourios.miner.* registry names; the dimensions shown are exported as attributes, not labels — tenant is ourios.tenant, service is ourios.service):

ourios.miner.confidence.p50 and ourios.miner.confidence.p01. The ourios.miner.confidence histogram is the source of truth; the two gauges are convenient named views derived from it in-process. The miner recomputes them on a short ticker (default 10 s, configurable; the cost is one quantile evaluation over the histogram per tenant per service per tick — negligible relative to the hot path) and caches the value between ticks so a metric export cycle never blocks on recomputation. The gauges exist so alerting rules and runbooks can name them directly per [§3.1] rather than spelling out a histogram_quantile(...) expression at every reference.

The histogram bucket boundaries are tuned to straddle the decision boundary at 1.0 (see §6.3): default buckets [0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 1.0, 1.05, 1.2, 1.5, 2.0, +Inf].

6.9 Persistence and recovery

Amendment 2026-06-10 (snapshot store / cadence / scope resolved). The three §9 open questions deferred from this section — target store, cadence, and scope — are now pinned. The snapshot is a rebuildable recovery-acceleration cache, not durable state (the WAL is the truth per [§3.4]), which is what licenses the choices: local disk, WAL-adjacent (object storage deferred, see §9); per WAL-segment-rotation cadence, recording the WAL high-water mark; per-tenant scope. The format is a leading u8 version byte then a payload; recovery dispatches on byte 0. In v1, recovery replays the full WAL in both the known- and unknown-version branches, because the resume-from- high-water-mark optimisation needs the RFC 0008 §6.7 checkpoint / replay-from-offset API, which is not yet implemented; the format records the high-water mark so the optimisation can be switched on later without a format change. This makes the §3.5.1 / §3.5.2 acceptance criteria buildable. The matching §9 entries are marked RESOLVED.

Amendment 2026-06-12 (v2 — restore switched on). The RFC 0008 §6.7 checkpoint / offset-carrying-sink API this section was gated on is now specified to land (RFC 0008’s same-day §6.1/§6.7 amendment is the other half of this design), so the known-version branch of the recovery algorithm below restores the tree and replays only the WAL tail above the snapshot’s recorded high-water mark S — exactly as written in step (2), with no format change (the mark has been in the payload since v1). Three rules complete the design. Per-consumer horizons: RFC 0008’s replay delivers every surviving frame with its offset; the recovery driver suppresses per consumer — the Parquet path consumes only frames above the RFC 0008 checkpoint X (below it they are already published), the miner only frames above S (below it they are already folded into the snapshot; re-feeding would double-apply — the v1 hazard, now resolved by routing rather than by refusing to restore). The rule covers both orderings: in the steady state S ≥ X the miner consumes a suffix of what Parquet consumes; with a lagging snapshot (S < X) the miner additionally consumes the (S, X] frames the floor retained, closing its state gap while Parquet suppresses them. Truncation floor: the ingester passes the latest durable snapshot’s S to Wal::housekeeping as a retain floor, so the WAL never unlinks a frame no snapshot has captured (RFC 0008 §6.7 — closes the S < X template-drift hole, hazard #5). Stale-snapshot fallback: if recovery nevertheless finds the WAL truncated past S (external mutation — segment files manually unlinked from wal_root; the floor prevents the gap arising internally), it restores the snapshot, replays the surviving frames above S, and emits a structured warning naming the gap. The data side is complete provided the truncation did not exceed X — legitimate housekeeping never unlinks a frame above the checkpoint, so everything missing is in Parquet; manual deletion beyond X would be unrecoverable acknowledged-data loss, which is exactly why segment removal is reserved to housekeeping and never an operator action. Templates first seen inside the (S, X] gap may re-mint (drift surfaced via RFC 0010, not silent). New acceptance criteria: §3.5.3 (restore-equivalence), §3.5.4 (stale-snapshot fallback); the end-to-end driver contract is RFC 0008’s RFC0008.10.

Hot path. The per-tenant tree lives in process memory on the ingester. Tree operations (descend, simSeq, attach, widen) are hot-path; persistence does not happen synchronously per line.

Durability via WAL replay. [§3.4] (WAL-before-ack) means every line that reached the miner is in the WAL before the ingester acknowledged it. The tree state is therefore derivable from the WAL: a cold start with no snapshot replays the WAL in order through the miner and reconstructs the trees. This is correct but slow at scale; the snapshot mechanism is an optimisation on top.

Replay mode. Cold-start replay re-walks attach, widen, and expand_slot_types against the same code path live ingest uses. Doing so naively would re-fire every counter increment, every histogram observation, and every gauge update for the entire replay window, polluting steady-state metrics for the post-restart horizon (a 10-minute replay on a high-volume tenant could shift ourios.miner.merges by orders of magnitude in a few seconds). The miner therefore runs in replay mode until the WAL cursor reaches the live tip: domain events are processed and tree state is mutated exactly as in live ingest, but updates to the §6.8 metrics are suppressed (counters do not increment, histograms do not observe, gauges retain their previous value or, if the miner has never served live traffic, their zero / empty initialisation value). Suppressing the update path means the replay window contributes no data points — each instrument still surfaces on its first live measurement once replay completes (§3.1.2’s registry-defined set is satisfied by instrument registration plus real post-replay traffic, not by replay-window points). A single wal_replay_progress gauge (attribute ourios.tenant, value: fraction of the tenant’s replay window completed in [0.0, 1.0]) is exposed during replay so operators can see the cold-start curve and confirm replay finished. This metric is replay-only and is not part of the §3.1 mandatory set; it is documented here, not in §6.8’s table.

Snapshot mechanism. A snapshot is a rebuildable recovery-acceleration cache, not durable state. The WAL is the durable truth ([§3.4]); the snapshot exists only to shorten cold-start replay. A lost, absent, or corrupt snapshot is never a data-loss event — it degrades to a full WAL replay (the same path a miner that never wrote a snapshot takes). This framing is what licenses the store and recovery choices below: because the snapshot is cache, it may live on local disk, and a reader that cannot trust it may discard it without ceremony.

Target store: local disk (WAL-adjacent); object storage deferred. Snapshots are written to a local artefact next to the WAL (e.g. under the WAL root), not to object storage. [§3.6] makes local disk legitimate here precisely because the snapshot is cache, not truth — the constraint [§3.6] imposes is that no feature rely on local disk being durable beyond the WAL horizon, and snapshot recovery never does: anything the snapshot would have accelerated is still in the WAL. Object-storage snapshots are explicit future work (see §9); they would couple to the RFC 0009 §3.4 atomic-publish manifest to define a durable, multi-writer publish point, and are deferred for that reason.

Scope: per-tenant. One snapshot artefact per tenant tree, matching [§3.7]’s per-tenant trees. Recovery loads the latest snapshot per tenant independently; there is no cluster-wide combined artefact.

Cadence: per WAL-segment rotation. A snapshot is taken at WAL-segment-rotation boundaries. The snapshot records the WAL high-water mark — the WalOffset (RFC 0008 §6.1) up to which its tree state reflects appended frames — so that a future optimisation can resume replay from there rather than from the start of the log.

Format: a leading u8 version byte, then the payload. Byte 0 is the snapshot format version; the remaining bytes are that version’s serialised payload. The payload captures the per-tenant state needed to reconstruct the miner: the tree leaves (template token sequence, template_id, template_version, the (severity_number, scope_name) template key of §6.1, and the per-slot slot_types of §6.1), the structured-template-id map allocated in §6.2’s structured short-circuit, and the WAL high-water mark above. The concrete payload codec is an implementation detail behind the version byte — the version byte is what makes format evolution safe, so this RFC pins the framing, the captured state, and the rule that the reader dispatches on byte 0, and deliberately does not pin a specific serialisation codec.

Recovery algorithm. On ingester restart, per tenant:

1. Load the latest snapshot artefact for the tenant, if one exists.
2. If byte 0 is a known version: deserialise the payload, restore the tree, then replay only the WAL tail above the snapshot’s recorded high-water mark S — the driver delivers each replayed frame to the miner only when its offset is > S (per-consumer routing, RFC 0008 §6.6; active as of the 2026-06-12 v2 amendment above). If the surviving WAL no longer reaches back to S, apply the stale-snapshot fallback of the v2 amendment: restore, replay what survives, warn.
3. If byte 0 is an unknown version, or the snapshot is absent or corrupt: discard it and replay the full WAL via Wal::replay (RFC 0008 §6.1 API, RFC 0008 §6.6 recovery procedure), rebuilding the tree from scratch.

v1 scope — rebuild from a full replay; do not restore yet. (Superseded by the 2026-06-12 v2 amendment above — restore is now switched on; this paragraph is retained as the record of why v1 refused to restore.) The restore-then-replay-the-tail path in step (2) requires the RFC 0008 §6.7 checkpoint / replay-from-offset API (Wal::checkpoint and the CHECKPOINT sidecar), which was not yet implemented at the time. Restoring a tree from a snapshot and then replaying the full WAL (the only replay available without offset support) would double-apply every frame the snapshot already captured, corrupting the tree. So until offset-resume landed, recovery ignored the snapshot payload and rebuilt the tree from a full Wal::replay in both branches — the known-version branch did not restore. What landed in v1 was the snapshot format (the leading version byte and the recorded high-water mark) and the version-dispatch + WAL-fallback contract; v2 switches the restore path on with no format change, exactly as planned. v1 fully satisfied §3.5.1 (the artefact carries a leading version byte) and §3.5.2 (an unknown version is rejected and falls back to full WAL replay).

Snapshot-load telemetry. The wal_replay_progress gauge (above) remains the replay-only signal. A snapshot-load-outcome signal — distinguishing “snapshot restored,” “unknown version → full replay,” and “absent/corrupt → full replay” — is named here in prose; its concrete metric and attribute names go through the semconv weaver registry when the slice is implemented (§3.1.2), and are not invented as flat names in this RFC.

Migration. When the in-memory data model in §6.1 changes (new field, retired field, semantic change), the snapshot format’s version byte increments. Old snapshots are read-compatible only if the change is additive (new optional fields tolerated). For breaking changes, snapshots from the prior version are discarded and the tree is rebuilt from WAL replay. [§3.5]’s schema-change discipline applies: the change goes through an RFC.

7. Alternatives considered

Alternatives to Drain itself, evaluated as primary algorithms. Each is rejected for the reason given; some have a possible secondary role noted.

Spell (LCS-based online parser)

Spell uses longest-common-subsequence to compare a new line against existing templates. Per-line cost is O(template_count × line_length) without depth bounding, which is several orders of magnitude slower than Drain’s O(d) tree walk at the template counts we expect (10²–10⁴ per tenant). LCS also makes parameter positions ambiguous on lines where the same token recurs, because the LCS alignment can shift; Drain’s positional matching gives unambiguous parameter slots. Rejected as the primary algorithm.

IPLoM (iterative partitioning)

IPLoM does three passes over the entire log, each splitting clusters by a different criterion (token count, position, token uniqueness). This requires the full log up front and is offline by design. Rejected as the primary algorithm. Possible secondary role: a periodic offline reconciliation pass could use IPLoM to detect template fragmentation that Drain’s online structure missed (e.g. two leaves that should have been one because their discriminating token was spurious). This is a follow-up RFC topic, not a §6 commitment.

LenMa (length-based clustering)

LenMa groups lines by token-count length, then finds templates within each length group via a similarity-based second pass. The length-only initial grouping is close to Drain’s first level, but the absence of the token-prefix tree leads to more spurious merges within a length group (any two same-length lines are candidates, not just same-length-and-same-prefix lines). Drain’s tree is a strict refinement of LenMa’s grouping. Rejected as the primary algorithm — Drain dominates on the same workload.

LogPPT / LILAC / LLM-based parsers

Transformer-based parsers achieve higher accuracy on benchmark corpora (LogPAI scores) but require model inference per line. At the D1 hot-path budget (≥ 100k lines/s/core), per-line transformer inference is infeasible without specialised hardware that contradicts §1’s “single Rust binary” framing. Rejected as the primary algorithm. Possible secondary role: offline labeling-aid on the testdata/corpus/ to bootstrap a labeled set for confidence calibration; or as a periodic reconciliation pass similar to IPLoM. Both are deferred to follow-up RFCs.

Offline clustering (e.g. nightly hierarchical agglomerative)

Quality is high; latency is unacceptable. Logs ingested at 14:00 would not be queryable until the next clustering window completes. This contradicts §2’s online motivation. Rejected as the primary algorithm. Possible secondary role: the same reconciliation pass mentioned under IPLoM and LLM-based could use offline clustering to validate Drain’s online output and surface drift; a follow-up RFC if and when reconciliation becomes a real concern.

8. Testing strategy

Mapping to [§6.2]. Each technique below names the §5 scenarios it operationalises; the test code carries the matching id in a doc comment per docs/verification.md §2.3 so grep -R "H1.1" . resolves bidirectionally between RFC and tests.

• Unit tests for tree operations: tokenize, mask, descend, simSeq, widen, attach, build_params. Each operation tested in isolation against fabricated inputs. Covers: RFC0001.3 (tokenizer whitespace-only), RFC0001.4 (confidence ratio + decision boundary), RFC0001.7 (combined widening + type-expansion in one attach).

• proptest for §6.6 reconstruction: for every generated line shape (length, separator distribution, masking outcome), reconstruct(mine(line)) == line or mine(line).lossy_flag == true. Property failure blocks merge. Covers: H7.1, H7.4, §3.3.1.

• Corpus tests on testdata/corpus/ (fixed, anonymised; see docs/benchmarks.md §1): assert bounds on ourios.miner.template.count, ourios.miner.merges, reconstruction accuracy, parameter overflow rate. Regressions are build failures, not warnings. Covers: H1.1 (login/logout corpus arm), H7.1 (corpus arm).

• Confidence calibration test: on a labelled subset of the corpus, verify the three-zone classification in §6.3 against the human labels. Covers: H1.2.

• Merge-audit assertion (negative + positive): no widening or type-expansion completes without a matching audit event, and fresh-leaf creation does not emit one. Runs on every corpus pass and on the synthetic widening fixtures. Covers: H1.3, H5.1, H5.2, RFC0001.1 (negative — no event on creation), RFC0001.2 (rejection event for degenerate widening), RFC0001.7 (event ordering arm).

• Multi-tenant isolation (negative test): interleave lines from two synthetic tenants through a single MinerCluster; assert that templates mined under tenant A never appear in tenant B’s tree and vice versa. Implements docs/benchmarks.md E2. Covers: §3.7.1, §3.7.2.

• Per-ResourceLogs tenant derivation (miner-side stub): assert that when records carrying distinct derived tenant_ids arrive in the same ingest sequence, each lands in its derived tenant’s tree. The receiver-side test — that the wire-decode layer actually derives tenant_id per ResourceLogs.resource rather than per ExportLogsServiceRequest — is owned by RFC 0003 (see RFC 0003 §6.3); RFC 0001 owns only the miner-side contract. Covers: §3.7.3.

• OTLP-aligned template-key tests: hand-curated OtlpLogRecord fixtures exercising the §6.1 Template-key composition tuple. Assert that varying only severity_number produces distinct template_ids, varying only scope_name produces distinct template_ids, the severity_number = 0 (UNSPECIFIED) and scope_name = None edge buckets are each their own key value, and body.kind != AnyValue::String short-circuits per §6.2 step 0 with the §6.1 sentinel confidence = 1.0, lossy_flag = false. The time_unix_nano round-trip is a small unit test against the §6.1 record schema. Covers: H1.4, H1.5, RFC0001.9, RFC0001.10, RFC0001.11.

• Drift detection test: ingest a corpus where a template deliberately drifts mid-stream; assert that the drift query in §6.7 returns the drifted template within the expected window. Covers: H5.3.

• Crash recovery test (snapshot + WAL replay): SIGKILL the ingester between snapshot writes; assert that recovery reconstructs the same tree state that was acknowledged before the kill. Also corrupt the snapshot’s leading version byte and assert WAL fallback. This is [§3.4]’s crash-recovery test extended to cover the miner’s persistence layer. Covers: §3.5.1, §3.5.2.

• Restore-equivalence test: snapshot a tree at high-water mark S, append further frames, recover via restore-plus-tail-replay, and assert tree-state equality against a from-scratch control (the recovered state is compared field-by-field via the §6.9 snapshot payload of both trees, so the comparison itself can’t hide drift). A counter on the test sink asserts no frame ≤ S reached the miner. The stale-snapshot arm deletes the segments holding (S, tail]’s prefix, recovers, and asserts the structured warning names the gap while the surviving frames still fold. Covers: §3.5.3, §3.5.4 (the end-to-end driver half is RFC 0008’s RFC0008.10).

• Configuration tests: assert default values and the rejection of out-of-bounds settings at startup. Covers: §3.1.1 (default threshold = 0.7), §3.2.1 (default param byte limit = 256), §3.2.2 (limit > 1 KiB rejected).

• Metric collection test: assert the mandatory set equals the generated ourios_semconv::OURIOS_MINER_* constants (the registry is the source of truth), then construct a miner, ingest a small representative workload that exercises every instrument, collect its meter (global::meter("ourios.miner")) via an SDK in-memory reader, and assert the collected stream contains every §6.8 metric name (each appearing on its first real measurement, with the required attributes), with the instrument kinds and attributes in §6.8’s table, and that the ourios.miner.confidence.p50 / ourios.miner.confidence.p01 gauges track the same-attributed ourios.miner.confidence histogram quantiles. Covers: §3.1.2, RFC0001.8.

• Data-model contract tests: small unit tests against the template_id query semantics that RFC 0002’s DSL compiles to. These cover the cross-version vs. cross-alias distinction at the data-model layer; the DSL surface itself is tested in RFC 0002. Covers: RFC0001.5, RFC0001.6.

• Alias write-path tests: assert that an operator alias assertion emits a durable alias_asserted audit event under the §3.4 barrier and that folding the event log produces the expected per-tenant alias map; that resolves_to expands by the set (representative or member) and to {X} for an un-aliased id; that a retraction emits alias_retracted and drops membership on rebuild; and that an alias asserted in one tenant is invisible to another. The resolves_to DSL surface itself is exercised in RFC 0002 (RFC0002.9); these tests own the §6.7 write-path and per-tenant-map contract. Covers: RFC0001.12, RFC0001.13, RFC0001.14, RFC0001.15, RFC0001.16.

• Reader behaviour test: assert the §6.6 Reader render contract — for a lossy_flag = true row the reader returns the body bytes verbatim (no in-band prefix/marker) carrying Reconstruction::RetainedVerbatim, and does not call reconstruct(); for a faithful row it calls reconstruct() and carries Reconstruction::Faithful. The reader never silently substitutes one rendering for the other, and never mutates the body bytes to carry the marker. (The clean-path read-time template-registry lookup is out of scope of this scenario per §6.6.) Covers: H7.3.

• Overflow-path tests: synthesize a parameter exceeding the configured byte limit; assert the OVERFLOW marker, forced body retention, and metric increments. Wire the alert-rule fixture for the >1% rate trigger. Covers: H2.1, H2.2.

• Tokenizer-failure tests: feed lines with embedded NULs, malformed UTF-8, and over-cap lengths; assert the parse-failure path retains the body and sets lossy_flag = true. Covers: H7.2.

• Benchmark (criterion): per-line miner latency (target: median ≤ 10 µs/line on the §1 hardware baseline), ingest throughput (target: ≥ 100k lines/s/core, per docs/benchmarks.md D1). No §5 scenario; satisfies thesis-gate D1 directly at the Validated stage.

9. Open questions

Decisions explicitly deferred. Each must be resolved before this RFC’s status flips to accepted.

Persistence (from §6.9) — RESOLVED (2026-06-10). The three sub-questions are pinned in §6.9; the resolutions are recorded here and the remaining future work is the two items below them.

• Snapshot target store — RESOLVED: local disk (WAL-adjacent); object storage deferred. The snapshot is a rebuildable recovery-acceleration cache, not durable state (the WAL is the truth per [§3.4]), so it lives next to the WAL on local disk. [§3.6] permits this because recovery never relies on the snapshot surviving — a lost snapshot degrades to a full WAL replay. See §6.9.
• Snapshot cadence — RESOLVED: per WAL-segment rotation. A snapshot is taken at segment-rotation boundaries and records the WAL high-water mark (the WalOffset it was taken at). See §6.9.
• Snapshot scope — RESOLVED: per-tenant. One snapshot artefact per tenant tree, matching [§3.7]; recovery loads the latest snapshot per tenant. See §6.9.
• Object-storage snapshots (remaining future work). Pushing snapshots to object storage would couple to the RFC 0009 §3.4 atomic-publish manifest for a durable, multi-writer publish point; deferred until that line settles.
• Resume-from-high-water-mark replay — RESOLVED (2026-06-12): switched on as §6.9 v2. The RFC 0008 §6.7 checkpoint / offset-carrying-sink API is specified (RFC 0008’s same-day amendment); the known-version branch restores and replays only the tail above the snapshot’s high-water mark, with per-consumer routing, a housekeeping retain floor, and a stale-snapshot fallback. No format change was needed — exactly as this entry predicted. Acceptance: §3.5.3 / §3.5.4 + RFC0008.10. See the §6.9 v2 amendment.

Algorithm tuning (open until corpus exists).

• Floor default 0.4 — confirm against the corpus. If the lossy zone is too wide (many lines retained that “should have” been parse failures), tighten; if too narrow (too many parse failures on lines a human would accept), loosen. This is per-tenant tunable; the question is the out-of-the-box default.
• Tree depth d. Paper default 4; Drain3 default 4. Open question: do any of our representative corpora benefit from d = 3 or d = 5?
• Max children per node. Drain3 caps at 100; the cap acts as a safety against unbounded fan-out from a bad masking rule. Confirm 100 is right for our corpora, or motivate a different number.

Edge cases.

• Lines that contain a literal <*> (the wildcard sentinel we use in template strings) — escape on tokenize, or replace with a non-collision character (e.g. U+E000)?
• Multi-line log entries (stack traces). Paper assumes single-line. Ourios position: deferred to RFC TBD on the OTLP receiver, since multi-line reassembly happens before the miner sees the line.

Multi-tenancy and operational lifecycle.

• Tenant lifecycle. §3.7 commits to per-tenant trees but does not name when a tree is allocated (lazily on first ingest? eagerly via a control-plane command?), nor whether tenants can be paused, evicted under memory pressure, or deleted. Likely deferred to a future operator-console RFC, but the bookend events (TenantInitialised, TenantPaused, TenantDeleted) need to exist somewhere before §3.7 is operationally complete.
• Per-tenant fairness and back-pressure. A noisy tenant can monopolise WAL bandwidth, blow up the tree, and starve well-behaved tenants. RFC 0001 has no rate-limit or back-pressure event in scope; this overlaps with the OTLP receiver’s responsibility and likely lives in a future ourios-ingester RFC.
• Alias index creation mechanism (from §6.7) — RESOLVED (2026-06-07). Of the three candidates (operator-driven, automatic-inference, deferred entirely), the maintainer chose operator-driven + audited: an alias is an explicit, audited, reversible operator assertion on the §6.4 stream, never silently inferred (§3.1). The write path, the alias_asserted / alias_retracted events, the per-tenant alias-map projection the querier reads, and the eventual-consistency semantics are specified in §6.7 (“Alias write path”); the acceptance criteria are RFC0001.12– RFC0001.16 in §5.3. RFC 0002’s template_id.resolves_to(X) now has a defined backing index. Remaining future work: automatic inference is deferred to a possible propose → operator-confirm layer (proposals never enter the active index unconfirmed); see the §6.7 “Future work” note. The physical alias-map file/format and snapshot cadence are a storage decision owned by the RFC 0005 line, not RFC 0001 — RFC 0001 owns the model, the write path, and the criteria (sibling to the issue #147 split). Update 2026-06-12: the RFC 0005 line has made the v1 half of that decision — alias events persist as kinds 4–5 on the RFC 0005 §3.7 audit stream, and the querier derives the map by folding that stream at compile time (RFC 0005 §3.7.1; §6.7 amendment of the same date). The cached per-tenant artifact remains deferred behind the RFC 0009 §3.4 manifest fork.

Cross-RFC contracts pending.

• Querier ↔ live template registry (from §6.6). Reconstruction’s lookup(template_id, template_version) is called by ourios-querier, which runs in a separate process from the ingester that owns the live tree. Candidates: querier reads snapshots from object storage (eventually consistent with live), querier asks the ingester via RPC at query time (couples query latency to ingester health), or templates ride a separate Parquet side-stream alongside records (eventually consistent, no RPC, new data plane). RFC 0002 needs the answer before its DSL can compile; this RFC names the seam.
• Audit-event Parquet schema (from §6.4). The Rust audit struct is specified in §6.4; the on-disk Parquet column layout for template_audit belongs to a future ourios-parquet RFC. The §6.7 drift query assumes the schema exposes event_type, template_id, old_version, new_version, and timestamp as columns suitable for predicate pushdown.

Deferred to follow-up RFCs.

• Reconciliation pass (IPLoM / offline clustering / LLM-based labeling) — if real-world drift turns out to be more than §6.4’s online widening can handle, a periodic offline pass becomes interesting. RFC at that point.
• Cross-tenant template_fingerprint side column — only if a concrete consumer materialises (storage dedup across tenants, shared dashboards). Until then, do not add.

10. References

• He, P., Zhu, J., Zheng, Z., Lyu, M.R. “Drain: An Online Log Parsing Approach with Fixed Depth Tree.” ICWS 2017.
• Drain3: https://github.com/logpai/Drain3 (specific commit pinned in this RFC at the Specified-gate PR).
• LogPAI logparser benchmark: https://github.com/logpai/logparser
• CLAUDE.md §§ 2, 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.7, 4, 6.2, 6.3, 6.6.
• docs/hazards.md H1, H2, H5, H7.
• docs/benchmarks.md C1, C2, C3, C4, D1, E1, E2.
• docs/rfcs/0002-query-dsl.md §5.4 (template primitives in the DSL surface; required to expose drift detection).
• docs/verification.md §§ 2, 3, 6 (the maturity model and the acceptance-criteria contract this RFC will inherit at the Specified gate).
• Future: docs/architecture/miner.md (this RFC graduates there on acceptance).

RFC 0002 — Query DSL

rfc: 0002 title: Query DSL — the Ourios logs query language (Branch B, surface β) status: green author: Jens Holdgaard Pedersen jens@holdgaard.org drafting-assistance: Claude created: 2026-04-24 supersedes: — superseded-by: —

RFC 0002 — Query DSL

Status note. The prior decision (§3) is resolved: the predicate sublanguage takes Branch B (distance from OTTL), on the β (pipe-composable) top-level surface. Decided 2026-06-07 from the audience analysis in §3.6 (primary: Perses dashboard authors; future: MCP agents). This RFC is now green — all 11 §5 acceptance criteria (RFC0002.1–.11) have passing tests (crates/ourios-querier/tests/rfc0002_dsl.rs), landed across PRs #143 (this spec) and #144–#154 (the red gate + implementation): the Branch-B parser + structured surface → one IR, the IR→DataFusion compile, YAML-embeddability + the structured JSON Schema, and resolves_to alias-set expansion via the RFC 0001 §6.7 operator alias map. §6 gives the design, §7 the grammar, §5 the criteria. Per the docs/rfcs/README.md ladder, validated and finally accepted (a maintainer flip) follow the §9 validation. Hazard 6 (CLAUDE.md §4 — no DataFusion/SQL leakage) constrains the whole design.

1. Summary

Ourios exposes a logs query DSL that does not leak DataFusion/SQL to users (CLAUDE.md §4 hazard 6). This RFC specifies it:

• Predicate sublanguage — Branch B (distance from OTTL). An Ourios-native, query-ergonomic syntax over the OTel data model (the ingest contract): bare top-level fields (body, severity, trace_id), resource. / attr. prefixes, bare-identifier severity (severity >= error), first-class template + OTel-canonical primitives.
• Top-level surface — β (pipe-composable). A predicate followed by pipe stages: … | range(-1h, now) | count by template_id | sort count desc | limit 10. Compact, single-line, and embeddable as a YAML scalar in Perses dashboards.
• Two front-ends, one core. The string DSL (for humans, esp. Perses YAML) and a structured JSON surface (for MCP agents + programmatic clients) parse to the same query IR and compile to the same DataFusion LogicalPlan. Agents emit JSON, not syntax.

The design rests on ourios-querier (RFC 0007), whose execution layer (predicate pushdown, tenant isolation, QueryStats) is already implemented and tested (RFC 0007 §5 criteria all live; the RFC itself stays specified pending this DSL); this RFC adds the user-facing language in front of it.

2. Motivation

2.1 Why a DSL at all?

CLAUDE.md §4 hazard 6 commits Ourios to a DSL that does not leak DataFusion SQL. The reasons are stability (evolve the backend without breaking user queries), safety (full SQL exposes cross-tenant joins, unbounded scans, recursive CTEs we cannot audit), and fit (logs are a narrow domain; a narrow DSL is more ergonomic than a general one). This is branch-agnostic.

2.2 Why the prior decision mattered

“OTTL-inspired” was not a free decision. Borrowing OTTL syntax in a query context promises OTTL-literate users that their mental model transfers; if the syntax looks the same but behaves differently (OTTL mutates; a query filters), the surface actively misleads. §3 records how that decision was made.

3. The prior decision (resolved): distance from OTTL

Both positions were defensible; §3.1–3.4 keep the honest case for each for the record. §3.5 records the resolution; §3.6 the reasoning.

3.1 The case for borrowing (Branch A) — not chosen

• Positive transfer for Collector-literate SREs. Engineers who write Collector/OTTL pipelines reuse that mental model at zero onboarding.
• Reduces bikeshed surface. A pinned external spec inherits decisions rather than re-litigating them.
• Ecosystem alignment. Diverging on surface syntax in the OTel orbit can read as gratuitous.
• OTTL’s path grammar is correct about the data model, which any alternative must address anyway.

3.2 The case for distancing (Branch B) — chosen

• The OTTL-literate population is a minority of OTLP users — most emit logs via an SDK and never touch OTTL.
• Collector ergonomics become query verbosity. resource.attributes["service.name"] == "api" is loud in a query.
• Shared syntax + different semantics misleads. Unfamiliar syntax is a safer failure mode than almost-familiar-but-wrong.
• No evolving external spec to track (OTTL has had breaking changes).
• Design freedom for query-context idioms (severity >= error, attr.foo).

3.3 What is shared regardless of branch

• The OTel data model is the schema of log records (the ingest contract, not a design choice): attributes, resources, severity, body, timestamps, trace context.
• The template + correctness primitives (template_id, confidence, lossy; drift-alias membership via resolves_to) are first-class (§6.3).
• The compilation target is a DataFusion LogicalPlan, no SQL leakage (§6.5).

3.4 Consequences

3.5 Resolution

Branch B (distance from OTTL), surface β (pipe-composable). Decided 2026-06-07 by the maintainer, on the audience analysis in §3.6 in lieu of the formal user research originally gated here (§9 now scopes that research to the accepted gate, not specified).

3.6 Why — the two audiences

The decision turns on two audiences that re-weight §3.1–3.4:

1. Primary — Perses dashboard authors (declarative YAML/CRDs). Queries live as string scalars in versioned YAML. Brevity and low bracket/quote density win (readable scalars, clean diffs); the audience thinks in dashboard query languages (PromQL/LogQL), not OTTL. Branch B’s flat syntax + the β pipe surface embed cleanly on one line; Branch A on surface α would be multi-line and bracket-heavy.
2. Future — MCP agents. Borrow-but-diverge (Branch A + the §6 divergence list it required) is the worst case for LLMs: strong public-OTTL priors pull a model toward real-OTTL constructs we do not support → plausible-but-invalid queries. A small, self-owned grammar (Branch B) has no priors to fight, is cheaper to embed in an MCP tool schema, and is enforceable with grammar-constrained decoding. And — decisively — agents need not generate syntax at all: they target the structured surface (§6.4).

The one strong case for Branch A (onboarding + signalling for Collector-literate SREs) lands on the audience that is not primary here, while its costs (semantic-confusion in the overlap zone; an externally-driven breaking cadence against long-lived dashboards + cached agent schemas) land squarely on these two. Distancing on surface syntax costs little ecosystem goodwill because we stay faithful to the OTel data model (§3.3) and because bespoke query syntax is the norm (LogQL, PromQL, CloudWatch Insights all diverge from any transformation language).

The full audience analysis is the drafting-assistance recommendation that informed this decision; its three OTel-ecosystem questions (is there an OTel query language to align with? is OTTL the expected querying surface? Perses+OTel query conventions?) are folded into §9.

4. Design principles

1. Familiarity beats cleverness. A first-time reader understands a query within 30 seconds without a reference. No heavy sigils.
2. No DataFusion/SQL leakage (CLAUDE.md §4 hazard 6). If explaining a surface form requires naming a DataFusion type, the form is wrong.
3. Predicate, then pipeline. A query is a predicate (the where) followed by ordered stages (range, aggregate, sort, limit, project). Each reads independently.
4. Template + OTel-canonical fields are first-class vocabulary, not pseudo-columns: template_id, confidence, lossy (drift-alias membership via resolves_to); service, trace_id, span_id, scope (the primary correlation/query dimensions per OTel maintainer guidance, §6.2).
5. Every query has a time range — explicit range(...) or a tenant-configurable default window. Never an unbounded scan.
6. One core, two surfaces. The string DSL and the structured surface are equivalent front-ends over one IR (§6.4); neither can express a query the other cannot.
7. YAML-embeddable. A query is expressible as a single-line scalar that survives a YAML round-trip — a first-class constraint for the Perses audience, not an afterthought.
8. The grammar is owned and versioned by this RFC (§7), not “inspired by” anything. Compatibility pledges are written, not implied.

5. Acceptance criteria

Given/When/Then, ids greppable from tests: each test carries the docs/verification.md §2.2 doc-comment form — /// Scenario RFC0002.<n> — <title>. plus /// See docs/rfcs/0002-query-dsl.md §5.. These specify the parser + compiler that front-ends the (already-implemented, RFC 0007 §5) execution layer.

• RFC0002.1 — A Branch-B predicate parses and compiles to a filter [CLAUDE.md §4 hazard 6]

  • Given a Branch-B predicate (e.g. template_id == 42 and severity >= error)
  • When it is parsed and compiled
  • Then it yields the query IR and an internal DataFusion Filter (a private compilation artifact — never surfaced through the public API, RFC0002.3). Predicates over RFC 0007 §4.3’s pushdown keys prune the scan per that section’s split — template_id skips row groups (B1), time_unix_nano prunes partitions and row groups, tenant_id prunes partition directories (not row groups); for the subset the current ourios_querier structured request can express (template + time) the DSL result is identical to it. Severity compiles via the §6.2/RFC0002.5 severity_number mapping (the column is RFC 0005’s severity_number), not the severity_text equality the current request supports, and predicates over non-indexed fields (service, attr.*) compile to a correct Filter with no row-group-pruning claim (indexed service.name pushdown would be a future RFC 0005 §3.6 amendment).

• RFC0002.2 — String DSL and structured surface compile to the same plan [§6.4]

  • Given a query expressed both as a β string and as the structured JSON surface
  • When both are compiled
  • Then they produce the same query IR (and hence the same LogicalPlan) — the one-core/two-surfaces invariant.

• RFC0002.3 — No DataFusion/arrow/SQL leakage [CLAUDE.md §4 hazard 6]

  • Given the public DSL API (parse, compile, error types)
  • When a query parses, compiles, or fails
  • Then no datafusion/arrow/SQL type or message appears in any public signature or error string (compile- and string-level boundary test, mirroring RFC0007.3).

• RFC0002.4 — A query without an explicit range gets the tenant default window [§4 P5]

  • Given a query with no range(...) stage
  • When it is compiled in a tenant context with a default window W
  • Then the plan carries a time-column filter equal to W — never an unbounded scan.

• RFC0002.5 — Bare-identifier severity maps to its SeverityNumber [§6.1]

  • Given severity >= error (and warn, info, debug, trace, fatal)
  • When compiled
  • Then each maps, case-insensitively, to the §6.1 SeverityNumber for that level (error → 17, etc.) and compiles identically to the numeric form (severity >= 17). The name→number mapping is the documented §6.1 one (Ourios’s, aligned with the OTel ranges) — not an OTel-standardised threshold.

• RFC0002.6 — First-class OTel-canonical fields resolve correctly [§6.2]

  • Given service, trace_id, span_id, scope used as bare fields
  • When compiled
  • Then each resolves to the RFC 0001 §6.1 column / resource-attribute it names (service → resource["service.name"]), with no string-flattening required of the user.

• RFC0002.7 — Parse/serialise round-trip is idempotent

  • Given any well-formed query (property-generated)
  • When parsed → serialised → parsed
  • Then the second parse equals the first (AST idempotence).

• RFC0002.8 — A malformed query yields a specific, leak-free error

  • Given a syntactically or semantically invalid query
  • When parsed/compiled
  • Then it returns a specific error citing the offending token/clause and the §7 grammar — never a panic, never a DataFusion message.

• RFC0002.9 — Template primitives compile [§6.3]

  • Given template_id == 42, resolves_to(42), lossy == true, confidence < 0.7
  • When compiled
  • Then each compiles to the documented plan (resolves_to expands to the alias-set membership of RFC 0001 §6.7), without leaking the underlying representation.

• RFC0002.10 — A query is a YAML-safe single-line scalar [§4 P7]

  • Given the canonical serialisation of any well-formed query
  • When embedded as a YAML scalar and round-tripped through a YAML parser
  • Then the recovered string parses to the same query (the Perses- embedding guarantee).

• RFC0002.11 — The structured surface validates against its published schema [§6.4]

  • Given the structured (MCP) query surface
  • When a request is validated against the published JSON schema
  • Then well-formed requests pass and compile; malformed ones are rejected by the schema before reaching the planner.

6. Design

6.1 Predicate sublanguage (Branch B)

A predicate is a boolean expression over paths, operators, and literals against the OTel log data model. The bare literal true is the match-all predicate (for queries that filter only by range/other stages); false matches nothing.

Paths.

• Top-level fields are bare identifiers mapping to the OTel log data-model fields: body (Body — an OTel AnyValue: string, bool, int, double, bytes, array, or kvlist/map), severity (SeverityNumber), ts (Timestamp), observed_ts (ObservedTimestamp), trace_id (TraceId), span_id (SpanId), scope (InstrumentationScope name), flags (TraceFlags). (Backend treatment of structured body vs attr.* is not uniform across the ecosystem; the DSL keeps the split explicit rather than flattening.)
• Resource attributes: resource.<key> where <key> is the OTel attribute key taken literally including dots (resource.service.name → resource attribute "service.name"). Bracketed form resource["..."] for any key not expressible as dotted bare identifiers — characters outside the bare-identifier set, a segment starting with a digit, or a reserved-word collision (resource["k8s.pod.name"], resource["3rd.party"]).
• Log-record attributes: attr.<key> (attr.http.status_code → attribute "http.status_code"); bracketed attr["..."] for the same non-bare-identifier cases.
• Severity: severity compares against a bare severity name (severity >= error), case-insensitive, or a numeric form (severity >= 17). All severity comparisons — including ordering (</<=/>/>=) — are defined on the OTel SeverityNumber, never on the free-form severity_text (per the OTel comparing severity guidance). Bare names map to the floor of the matching OTel SeverityNumber range: trace→1, debug→5, info→9, warn→13, error→17, fatal→21. The spec standardises the ranges and says to compare on SeverityNumber; this name→number mapping is Ourios’s, aligned with those ranges, not separately mandated by OTel.

Operators. Comparison: ==, !=, <, <=, >, >=, =~ (regex match), !~ (regex non-match). Boolean: and, or, not, with terse aliases &&, ||, !; grouping with ().

Literals. Double-quoted strings ("api"), numbers (500, 0.7), booleans (true/false), null, duration literals (30s, 1h, 1d, 1w), and RFC 3339 timestamps.

Functions (read-only, bespoke names tuned for queries) — boolean predicate terms: matches(path, regex), contains(path, s), starts_with(path, s), ends_with(path, s). They require a string operand: applying one to a non-string path (severity, a numeric/bool attribute, lossy, ts) is a compile-time type error (RFC0002.8), not a silent coercion. (Scalar-returning functions such as len(path) are deferred: the grammar admits a call only as a boolean term, so a numeric len(...) > n would need a scalar-comparison form — added under a future minor version when a need surfaces.)

Worked predicate.

service == "api" and severity >= error and attr.http.status_code == 500

6.2 First-class OTel-canonical fields

Per OpenTelemetry maintainer guidance (the primary dimensions a log backend is judged on), these get named, bare surface rather than hand-written attribute lookups, resolving the last open question of the prior draft:

Amendment 2026-06-11 — range(...) filters the effective timestamp. This table previously noted that ts / time_unix_nano is “what range(...) filters”. Per RFC 0005 §3.2 (amendment of the same date), the time window shall compile against the derived effective_time_unix_nano column — time_unix_nano when non-zero, else observed_time_unix_nano.unwrap_or(0) (RFC 0005 §3.2 is the normative derivation; a record with neither timestamp stays at 0). The implementing slice follows this amendment; until it lands, the querier filters time_unix_nano directly. The change makes records whose source timestamp is unknown (time_unix_nano = 0 — ~15 % of real OTel-Demo corpora, per the OTLP logs data model’s “Use Timestamp if it is present, otherwise use ObservedTimestamp” recommendation) addressable by time. The bare ts field is unchanged — it still resolves to time_unix_nano, the verbatim wire value (RFC 0001 scenario RFC0001.10). For files written before the column existed the window applies effective := time_unix_nano (the RFC 0005 §3.9 documented default — exactly the pre-amendment behaviour), not the absent-OPTIONAL-column ⇒ predicate-false convention. The window bounds are half-open — range(from, to) selects from <= effective < to. The half-open shape is what the querier already implements today (over time_unix_nano) and matches RFC 0010’s locally-pinned [from, to) (which noted this RFC had not pinned boundary semantics; it now does).

trace_id / span_id literals are hex strings (32 and 16 hex digits respectively, no separators), parsed case-insensitively so uppercase OTLP/JSON ids are accepted; the canonical/serialised form is lowercase. The compiler hex-decodes them to match the stored byte columns — the OTLP/JSON id convention, consistent with RFC0003.6.

6.3 Template + correctness primitives

First-class vocabulary — Ourios-specific extensions (RFC 0001 §6.3/§6.7), not OpenTelemetry log-data-model fields; they live in the Ourios schema + query layer alongside the OTel-canonical fields of §6.2:

• template_id == 42 — exact template; resolves to the template_id column.
• resolves_to(42) — X plus its drift aliases (the RFC 0001 §6.7 drift question); compiles to alias-set membership over template_id.
• confidence — miner confidence (e.g. < 0.7); the confidence column.
• lossy — the lossy-reconstruction flag; resolves to the RFC 0001 / RFC 0005 lossy_flag column (lossy == true).
• render (pipe stage, §6.5) reconstructs the original line, honouring lossy.

The drift question is answered by resolves_to (alias membership). A bare drift predicate (“has this template drifted?”) is deferred: per RFC 0001 §6.7 drift is an audit-stream property, not a column in the RFC 0005 data files, so it needs an audit-stream query path — a future capability, not a row predicate in this grammar.

6.4 Two front-ends, one core

flowchart LR
  A["string DSL (β)<br/>Perses YAML, humans"] --> P[parser]
  B["structured surface<br/>JSON, MCP agents + clients"] --> V[schema validate]
  P --> IR[query IR]
  V --> IR
  IR --> C["compiler<br/>(no SQL leakage)"]
  C --> LP["DataFusion LogicalPlan<br/>(RFC 0007 execution layer)"]

• String DSL (surface β) is the human surface (esp. Perses YAML). A query is a predicate optionally followed by |-separated stages:

  service == "api" and severity >= error | range(-1h, now) | count by template_id | sort count desc | limit 10
  

  A predicate-only / “no filter” query uses the match-all atom true, e.g. true | range(-1h, now) | limit 100.

  Stages: range(from, to) (each bound a relative duration, the now keyword, or an RFC 3339 timestamp — the §7 time form; defaults per §4 P5), count [by <field, …>] (comma-separated, per the §7 field_list) and other aggregations (sum, min, max, avg over a path), sort <field-or-aggregate> [asc|desc] (the §7 sort_key — a field or an aggregate output like count), limit <n>, project <field, …> / render. The whole query is expressible on one line as shown (whitespace around | is optional) — the §4 P7 YAML constraint.

• Structured surface is the machine contract (MCP tool schema + programmatic clients): a top-level object { "predicate": <node>, "stages": [ <stage>, … ] } (stages optional, default []). A field is structured (no DSL path syntax for agents to build or escape): a bare top-level name string ("service", "severity", "body", "trace_id", …) or an attribute object { "resource": "<key>" } / { "attr": "<key>" } (<key> the raw OTel attribute key, e.g. "k8s.pod.name"). An op is a §7 cmp_op string ("==", ">=", "=~", …); a value is a JSON primitive (string / number / bool / null), with durations and timestamps carried as their §7 lexical strings ("1h", RFC 3339). A <node> is a comparison node { "field": …, "op": …, "value": … }, a call node { "call": "<fn>", "args": [ … ] } whose args follow the §7 typed signatures — matches/contains/starts_with/ends_with take [ <field>, <string> ] (<field> as above), resolves_to takes [ <number> ], a constant node { "const": true | false } (the §7 bool_lit match-all / match-none — { "const": true } is the “no filter” predicate), or a boolean node ({ "and": [ <node>, … ] } / { "or": [ <node>, … ] } with a child array; { "not": <node> } unary, per §7). Each <stage> is a tagged object covering the full §7 stage set — range/count/sum/min/max/avg/sort/limit/project/render. Its JSON Schema is published and versioned with the parser (snapshot- tested like the §7 grammar; RFC0002.11), and it compiles to the same IR as the string surface (RFC0002.2). It is the formalised, extended successor to the existing ourios_querier::QueryRequest (the RFC 0007 structured API) and is the stable surface agents target — no grammar generation required.

Both parse/validate to the same query IR and compile identically (RFC0002.2). The tenant is not expressed in either surface — it is supplied by the executing context (CLAUDE.md §3.7 multi-tenancy; enforced per RFC0007.5); a query without a tenant is an API usage error, not a cross-tenant scan.

6.5 Compilation target

Every construct compiles to a DataFusion LogicalPlan:

All but render and resolves_to are DataFusion’s built-in algebra; those two are the only Ourios extensions, both surface-independent.

6.6 Stability and versioning

The grammar (§7) is owned and versioned by this RFC. Additions (new functions, new first-class fields) are minor versions. Behavioural changes that could alter a query’s result set are major versions, require an amending RFC + a deprecation window, and — because the Perses/MCP audiences persist queries (git-versioned dashboards, cached agent schemas) — ship with a documented migration. There is no external spec to shadow, so major versions are deliberate, not inherited.

7. Grammar specification (owned by this RFC)

A compact EBNF; the canonical machine-readable grammar lives beside the parser and is snapshot-tested (§8). Kept small and regular so it doubles as a constrained-decoding grammar for the MCP surface (§3.6).

query        = predicate , { "|" , stage } ;
predicate    = or_expr ;
or_expr      = and_expr , { ("or" | "||") , and_expr } ;
and_expr     = unary , { ("and" | "&&") , unary } ;
unary        = [ "not" | "!" ] , ( comparison | call | bool_lit | "(" , predicate , ")" ) ;
bool_lit     = "true" | "false" ;   (* match-all / match-none; a bare `true` = no filter *)
comparison   = severity_cmp | scalar_cmp ;
severity_cmp = "severity" , ord_op , ( severity_name | number ) ;
scalar_cmp   = scalar_path , cmp_op , literal ;
ord_op       = "==" | "!=" | "<" | "<=" | ">" | ">=" ;   (* no regex — severity is numeric *)
cmp_op       = ord_op | "=~" | "!~" ;
call         = str_fn , "(" , path , "," , string , ")"
             | "resolves_to" , "(" , number , ")" ;
str_fn       = "matches" | "contains" | "starts_with" | "ends_with" ;
path         = field | "resource" , key_tail | "attr" , key_tail ;
scalar_path  = nonsev_field | "resource" , key_tail | "attr" , key_tail ;
field        = nonsev_field | "severity" ;
nonsev_field = "body" | "ts" | "observed_ts" | "trace_id" | "span_id"
             | "scope" | "flags" | "service" | "template_id"
             | "confidence" | "lossy" ;
key_tail     = ( "." , dotted_key ) | ( "[" , string , "]" ) ;
dotted_key   = ident , { "." , ident } ;
stage        = "range" , "(" , time , "," , time , ")"
             | "count" , [ "by" , field_list ]
             | agg_fn , "(" , path , ")" , [ "by" , field_list ]
             | "sort" , sort_key , [ "asc" | "desc" ]
             | "limit" , integer
             | "project" , field_list
             | "render" ;
agg_fn       = "sum" | "min" | "max" | "avg" ;
field_list   = field , { "," , field } ;
sort_key     = field | ident ;          (* ident = an aggregate output, e.g. count *)
literal      = string | number | boolean | "null" | duration | timestamp ;
severity_name = "trace" | "debug" | "info" | "warn" | "error" | "fatal" ;  (* case-insensitive; only as a `severity` RHS *)
time         = "now" | ( [ "-" ] , duration ) | timestamp ;   (* e.g. now , -1h *)
integer      = digit , { digit } ;
(* lexical: ident = letter , { letter | digit | "_" } ;
   string = '"' , { char | escape } , '"' ;
   char   = any Unicode scalar except '"' , '\' , or a line terminator
            (a literal newline must be written as the \n escape — queries
            are single-line, §4 P7 / RFC0002.10) ;
   escape = '\' , ( '"' | '\' | "n" | "t" | "r" | ( "u" , 4 * hex ) ) ;
   number = integer | float ;  float = integer , "." , digit , { digit } ;
   boolean = "true" | "false" ;
   duration = integer , ( "s"|"m"|"h"|"d"|"w" ) ;  timestamp = RFC 3339 ;
   digit = "0".."9" ;  letter = "a".."z" | "A".."Z" ;
   hex = digit | "a".."f" | "A".."F"
   — strings are double-quoted with backslash escapes; YAML embedding
   (RFC0002.10) wraps the whole query in a single-quoted YAML scalar so
   these double quotes need no YAML-level escaping *)

8. Testing strategy

Mapping to CLAUDE.md §6.2 and docs/verification.md §3 (red→green two-loop: #[ignore]’d stubs first, implementations second).

• Unit tests — every grammar production has a positive and negative parse test.
• Property tests — generate well-formed queries; assert the §5 round-trip idempotence (RFC0002.7) and that every generated query is a YAML-safe single-line scalar (RFC0002.10).
• Compilation golden tests — every construct has a golden LogicalPlan (debug-rendered) checked in; the no-leakage boundary (RFC0002.3) is a compile + string test.
• Equivalence tests — string vs structured surface compile to the same IR (RFC0002.2); a DSL query and the equivalent ourios_querier structured request return identical results + QueryStats (RFC0002.1).
• Grammar snapshot — the EBNF / parser grammar is committed and snapshot-tested so changes are PR-visible (Branch B owns its grammar).
• End-to-end — against the docs/benchmarks.md §1 corpora, pinned expected results for a query set spanning each construct.

9. Open questions

Narrowed by the §3 resolution. Must be resolved before accepted.

• Pre-accepted validation. The §3.6 audience analysis stands in for instinct, not for evidence: before accepted, run a readability pass on 10–20 sample queries with non-author reviewers, and a migration sketch from LogQL/Insights into β. (Replaces the prior §9 user-research gate; not required for specified.)
• OTel ecosystem alignment Resolved: OpenTelemetry defines the logs data model + API/SDK but no standard query/read language, and OTTL is a Collector transformation language, not a querying surface (see the OTTL README and the OTel logs spec linked in §11 References). There is no canonical OTel read syntax, and no Perses-specific query convention, to align to. Bespoke query syntax over the OTel data model is the norm (LogQL, PromQL, CloudWatch Insights), so Branch B carries no ecosystem-divergence cost — the alignment that matters is at the field semantics, which §6.1/§6.2 honour (ts/observed_ts/trace_id/span_id/flags/body/scope/ severity → canonical data-model fields; severity ordering on SeverityNumber).
• --sql advanced-mode escape hatch — gated + sandboxed, or never? (Currently: never; reconsider under a separate RFC.)
• Custom user functions — out for v1 (sandboxing is its own project).
• params[N] positional access vs named parameters via the template schema.
• In-path query cost estimator (“this will scan 400 GB”) before run.
• Pagination / streaming surface for large result sets (mirrors RFC 0007 §8).

Resolved by this RFC (were open in the draft): branch (B), top-level surface (β), severity-text casing (case-insensitive, §6.1), agent- friendliness (the structured surface, §6.4), and first-class OTel- canonical fields (§6.2).

10. Alternatives considered

Alternatives that would replace the whole design, not just one branch.

• Pure SQL (DataFusion dialect) — zero parser cost, but violates CLAUDE.md §4 hazard 6 (cross-tenant joins, unbounded scans) and binds the user surface to DataFusion. Rejected as default; possible future gated, sandboxed escape hatch under a separate RFC.
• LogQL clone — label selectors are less expressive than the OTel log record; adopting them flattens structure and lies about the ingest contract. Rejected as the full DSL; its top-level shape survives as the chosen β surface.
• CloudWatch Insights clone — proprietary, no open spec; attribute model differs from OTel. Rejected; its verb-per-line readability is the γ alternative we did not pick.
• Branch A (borrow OTTL) on any surface — see §3; not chosen for the Perses/MCP audiences.

11. References

• OpenTelemetry log data model: https://opentelemetry.io/docs/specs/otel/logs/data-model/
• OpenTelemetry severity text conventions: https://opentelemetry.io/docs/specs/otel/logs/data-model/#field-severitytext
• OTTL (reference-only under Branch B): https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl
• LogQL: https://grafana.com/docs/loki/latest/query/
• CloudWatch Logs Insights: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html
• Perses (CNCF dashboards-as-code): https://perses.dev/
• Apache DataFusion logical-plan documentation.
• RFC 0001 §6.1/§6.3/§6.7 (the columns + template/drift primitives); RFC 0007 (the execution layer this DSL targets); CLAUDE.md §4 hazard 6 (no-leakage hazard) and CLAUDE.md §3.7 (multi-tenancy).

RFC 0003 — OTLP receiver

rfc: 0003 title: OTLP receiver — gRPC and HTTP wire endpoints for OpenTelemetry log ingest status: green author: Jens Holdgaard Pedersen jens@holdgaard.org drafting-assistance: Claude created: 2026-05-13 supersedes: — superseded-by: —

RFC 0003 — OTLP receiver

How to read this document. Sections §§1–4 are the design contract — the what and the why. §5 lists the normative Given / When / Then scenarios — the contract the receiver crate is implemented against and tested for. §6 is the precise specification the receiver crate is implemented against. §7 records the alternatives we evaluated and rejected. §8 maps each §5 scenario to the technique that tests it. §9 lists open questions; §10 the references.

Cross-references to CLAUDE.md sections are in square brackets, e.g. [§3.4], and name the invariant the section must preserve. Cross-references to RFC 0001 use its section numbers directly (e.g. RFC 0001 §6.1).

1. Summary

The Ourios OTLP receiver accepts OpenTelemetry log batches over gRPC and HTTP, decodes them per the official opentelemetry-proto schema, derives a tenant_id per ResourceLogs group via an operator-configured rule (RFC 0001 §6.1 Tenant derivation), materialises each LogRecord.body into the Body::String(String) | Body::Structured(AnyValue) fork (the decoded AnyValue rides through verbatim — canonicalisation happens once, at ingest, inside the miner, per the amended §6.4), fans the batch out into per-tenant streams of OtlpLogRecord, hands each stream to ourios-miner, and acknowledges the OTLP request only after the WAL-before-ack invariant [§3.4] is satisfied. The default wire stack is tonic (gRPC) + axum/hyper (HTTP) against the official opentelemetry-proto Rust crate; the alternatives considered and rejected are embedding rotel as a library and running the OTel Collector out-of-process.

This RFC is the wire-decode contract that the §6.1 amendment of RFC 0001 (PR #21) and the §6.2 algorithm rewrite (PR #23) both implicitly require: the miner takes a structured OtlpLogRecord that something must produce. RFC 0003 is that something.

2. Motivation

2.1 The OTel-native commitment is not yet implemented

docs/glossary.md (entry OTLP) commits Ourios to OTLP as the sole ingest contract: “we do not invent our own format.” RFC 0001’s pre-amendment §6.1 record schema and the MinerCluster::ingest(_, raw: &str) signature treated logs as flat text strings, which the investigation in docs/architecture/otlp-log-format.md (PR #20) showed to be incompatible with that commitment. PRs #21 and #23 amended the miner’s data model and algorithm to consume structured OtlpLogRecords. No code yet produces those records. This RFC specifies the producer.

2.2 The receiver is the boundary that decides what “OTLP” means in practice

OTLP carries a structured LogRecord whose body is AnyValue (string, bool, int, double, bytes, array, kvlist), whose attributes are typed, whose Resource lives one container level up (per ResourceLogs), and whose timestamps and severity are first-class. Where in the pipeline these wire-level facts become the in-memory OtlpLogRecord the miner sees is a load-bearing decision: the receiver is where:

• The wire format (protobuf vs JSON, gRPC vs HTTP) collapses to a single in-memory representation.
• tenant_id is derived per ResourceLogs (RFC 0001 §6.1 Tenant derivation) and the batch fans out into per-tenant streams.
• body.kind = Structured records have their AnyValue body carried verbatim into Body::Structured(AnyValue) — per the amended §6.4 the receiver never canonicalises; the miner encodes the tree to the Ourios canonical body encoding at ingest, whose round-trip stored_bytes ↔ AnyValue per RFC 0001 §6.1 Body representation makes the lossy_flag = false promise meetable.
• The acknowledgement-after-durability sequencing ([§3.4]) is enforced.

Specifying these decisions in one place — and pinning them explicitly against the OTel spec rather than reinventing them — is what this RFC does.

2.3 Roadmap context

docs/roadmap.md §5 (post-#22) lists “OTLP wire endpoints (gRPC + HTTP listeners)” as post-MVP: the bench reads OTLP from disk (a corpus of pre-recorded LogsData), not from the network, so wire-decode is not on the C2 thesis-gate path. The record shape is in MVP — the miner consumes OtlpLogRecord from the corpus reader. This RFC is the spec for the post-MVP wire layer; landing the spec now (rather than after the bench) settles the design while the OTLP record shape is being implemented in the miner, so the receiver’s eventual implementation has nothing to redesign.

3. Background — OTLP wire formats

3.1 The OTLP message hierarchy

An OTLP log export is a single ExportLogsServiceRequest message carrying one or more ResourceLogs. (LogsData is the file-format equivalent message in logs.proto and shares the same resource_logs: ResourceLogs[] field shape; this RFC uses ExportLogsServiceRequest throughout, since that is the wire type both transports decode into.)

ExportLogsServiceRequest
└── resource_logs: ResourceLogs[]
    ├── resource: Resource           # per-source attributes (service.name, host.*, ...)
    ├── schema_url: string
    └── scope_logs: ScopeLogs[]
        ├── scope: InstrumentationScope     # name, version, attributes
        ├── schema_url: string
        └── log_records: LogRecord[]        # the actual log entries

A single export request can carry records from multiple sources (multiple ResourceLogs groups), each with its own Resource, and within each Resource multiple instrumentation scopes. The mapping from this hierarchy to per-tenant streams of records is the receiver’s responsibility (§6.4 below).

3.2 Two transports, three encodings

OTLP is defined for two transports:

• OTLP/gRPC — the canonical transport. Service is opentelemetry.proto.collector.logs.v1.LogsService, method Export. Wire encoding: protobuf over HTTP/2.
• OTLP/HTTP — POST against the /v1/logs path. Wire encoding chosen by the client per the Content-Type header:
  • application/x-protobuf (recommended by the spec; the same protobuf message as gRPC)
  • application/json (the proto3 JSON mapping with OTLP overrides — hex trace_id/span_id, base64 bytes)