Ourios

οὔριος — the fair wind that fills a ship’s sail. The mark is the thesis as a glyph: the sail, and the same shape mirrored as log lines — the data is the wind.

Ourios is a log storage and query backend built on Apache Parquet, a Drain-derived online template miner, and Apache DataFusion. The thesis: Parquet + template mining + DataFusion collapses the inverted index, the compression layer, the storage tier, and the query engine into one stack of off-the-shelf parts plus thin glue. Log lines collapse to (template_id, params) at ingest — a logical 50–200× reduction whose payoff is query pruning (a selective query reads a handful of row groups instead of scanning the corpus), not on-disk bytes that beat a byte codec. Our job is the glue, plus the honest handling of the places where template mining can go wrong.

The repository README is the short front door; this book is the design record and the depth behind it.

Project status

Pre-release, under active RFC-driven development. The full path — OTLP in; WAL, miner, Parquet on object storage; logs-DSL / DataFusion queries out — is implemented and tested behind RFC acceptance gates, and the performance thesis is measured (including against Grafana Loki) in Benchmarks. Signed pre-release binaries, container images, and a Helm chart exist, but interfaces and the on-disk schema can still move: treat everything as pre-1.0.

How this book is organised

• Getting started — run the single binary, configure it, deploy it with Docker or Helm, and lock it down before a listener leaves localhost.
• Architecture — the load-bearing reading: OTLP’s log data model vs. the miner’s view of it, hazards (where projects in this space die, and how we won’t), verification (how an RFC criterion becomes a red-gate test becomes a green one), and the glossary.
• Benchmarks — the thesis gates, stated so they could falsify the project, and every measurement against them with run IDs and caveats; plus the roadmap.
• RFCs — every subsystem is specified before it is built. Each RFC pins Given / When / Then acceptance scenarios and climbs a maturity ladder (drafted → specified → red → green → validated → accepted); the frontmatter status tells you how much to trust it.
• Talks — lecture-length explanations of the ideas behind the RFCs, for when you want the background rather than the specification.

Contributions, RFC discussion, and push-back on the invariants are all welcome — CONTRIBUTING.md and CLAUDE.md in the repository root carry the governing conventions, including the project’s intentionally fully AI-assisted development process under human maintainer review.

Quickstart — single binary

The fastest path from zero to querying logs: one ourios-server process on your machine, local-disk storage, no auth. This is the development/evaluation posture — see Kubernetes (Helm) for the production topology and Authentication before exposing any listener beyond localhost.

1. Get the binary

Download a signed release archive (Linux; Apple-silicon and Intel macOS builds come from cargo build today):

curl -LO https://github.com/jensholdgaard/ourios/releases/latest/download/ourios-server-x86_64-unknown-linux-gnu.tar.xz
tar -xf ourios-server-x86_64-unknown-linux-gnu.tar.xz

Releases from v0.1.1 on attach offline provenance bundles (*.intoto.jsonl) alongside their assets, verifiable without any network round-trip:

gh attestation verify ourios-server-x86_64-unknown-linux-gnu.tar.xz \
  --repo jensholdgaard/ourios \
  --bundle ourios-server-x86_64-unknown-linux-gnu.intoto.jsonl

Or build from source with cargo build --release -p ourios-server.

2. Run it

The binary is one server with three roles — receiver (OTLP ingest), querier (the logs-DSL API), and the background compactor (on by default). Enable the two network roles and point everything at a scratch directory:

mkdir -p /tmp/ourios/data /tmp/ourios/wal

OURIOS_BUCKET_ROOT=/tmp/ourios/data \
OURIOS_WAL_ROOT=/tmp/ourios/wal \
OURIOS_RECEIVER_ENABLED=1 \
OURIOS_RECEIVER_GRPC_ADDR=127.0.0.1:4317 \
OURIOS_RECEIVER_HTTP_ADDR=127.0.0.1:4318 \
OURIOS_QUERIER_ENABLED=1 \
OURIOS_QUERIER_HTTP_ADDR=127.0.0.1:4319 \
./ourios-server

Startup prints the bound addresses and warns once that auth is in open mode:

receiver gRPC listening on 127.0.0.1:4317
receiver HTTP listening on 127.0.0.1:4318
querier HTTP listening on 127.0.0.1:4319

The ports are the OTLP defaults (4317 gRPC, 4318 HTTP) plus 4319 for the query API; the explicit 127.0.0.1 binds keep this quickstart on localhost (the server defaults to 0.0.0.0 for container use). Prefer a config file over env vars? See Configuration — --config ourios.yaml makes the file the sole source.

3. Send logs

Ourios speaks OTLP and nothing else — any OpenTelemetry SDK or Collector can ship to it unmodified. Every export names its tenant out of band — the X-Ourios-Tenant header (HTTP) or x-ourios-tenant metadata (gRPC); resource attributes such as service.name describe the producer and never choose the tenant (RFC 0046).

With a Collector, point the OTLP exporter at it and set the tenant once:

exporters:
  otlp:
    # host:port is version-proof for the gRPC exporter; recent
    # Collectors also accept scheme'd forms.
    endpoint: localhost:4317
    headers:
      x-ourios-tenant: checkout
    tls:
      insecure: true

An SDK sets the same header through the standard variable: OTEL_EXPORTER_OTLP_HEADERS=x-ourios-tenant=checkout.

Or hand-deliver one OTLP/JSON record for a first smoke test:

curl -s http://localhost:4318/v1/logs \
  -H 'X-Ourios-Tenant: checkout' \
  -H 'Content-Type: application/json' \
  -d '{
    "resourceLogs": [{
      "resource": { "attributes": [
        { "key": "service.name", "value": { "stringValue": "checkout" } }
      ]},
      "scopeLogs": [{ "logRecords": [{
        "timeUnixNano": "1751971200000000000",
        "severityNumber": 9,
        "body": { "stringValue": "user 42 logged in" }
      }]}]
    }]
  }'

An empty {} response is the OTLP success shape. The batch is fsynced to the write-ahead log before that acknowledgement — kill the process mid-ingest and acknowledged data survives.

4. Query

POST /v1/query takes the logs DSL as plain text, with the tenant in a header:

curl -s http://localhost:4319/v1/query \
  -H 'X-Ourios-Tenant: checkout' \
  -H 'Content-Type: text/plain' \
  -d 'severity >= info | limit 10'

The response carries the total match count, the returned rows (bodies reconstructed from their mined templates), and scan statistics that show the Parquet pruning at work:

{
  "rows": 1,
  "stats": { "row_groups_scanned": 1, "row_groups_pruned": 0, "bytes_read": 4096 },
  "records": [ {
    "time_unix_nano": 1751971200000000000,
    "severity_number": 9,
    "body": { "kind": "rendered", "line": "user 42 logged in", "reconstruction": "faithful" },
    "...": "..."
  } ]
}

The DSL’s full grammar — field predicates, regex, time ranges, aggregation pipelines like service == "api" and severity >= error | count by template_id — is specified in RFC 0002.

Where to next

• Docker — the same server from the published image.
• Kubernetes (Helm) — the production topology on S3-compatible object storage.
• Authentication — static bearer tokens and OIDC; do this before any listener leaves localhost.
• The MCP surface (agents querying Ourios over the Model Context Protocol) rides the querier at /mcp — enable with OURIOS_QUERIER_MCP_ENABLED=1 (RFC 0027).
• Observe your coding agent — point Claude Code or Copilot CLI at Ourios and query its own cost and tool use back through the MCP surface, all on your machine.

Configuration

Two mutually exclusive sources (RFC 0020 / RFC 0004):

• A YAML file via --config <path> — the file is then the sole source; the environment participates only through ${env:NAME} substitution inside it (with ${env:NAME:-default} defaults, $$ escaping — the OTel Collector data model).
• OURIOS_* environment variables when no --config is given — the container/dev posture.

Parsing is strict: a malformed value is a startup error in either mode, and in config-file mode an unknown YAML key is rejected too (unrecognised OURIOS_*-lookalike env vars are simply not read — there is no unknown-key concept in the environment).

A complete file example

storage:
  # local (a filesystem directory as the store — dev/single-node) or
  # s3 (object storage as the source of truth — production; RFC 0019).
  backend: s3
  s3:
    bucket: ourios-logs
    region: eu-central-1
    # Any S3-compatible provider: AWS, MinIO, R2, Ceph/RGW, …
    endpoint: https://s3.eu-central-1.amazonaws.com
    # Secret hygiene is enforced: credentials MUST be ${env:…}
    # references — inline literals fail startup.
    access_key_id: ${env:OURIOS_S3_ACCESS_KEY_ID}
    secret_access_key: ${env:OURIOS_S3_SECRET_ACCESS_KEY}
  # RFC 0022: per-key promoted attribute columns (service.name is
  # always promoted). Each key costs bytes on every row — opt in
  # deliberately.
  promoted_attributes:
    resource: [k8s.namespace.name]
    log: [http.request.method, http.route]

receiver:
  enabled: true
  grpc_addr: 0.0.0.0:4317
  http_addr: 0.0.0.0:4318
  # The WAL stays on local disk by design, S3 or not (RFC 0019).
  wal_root: /var/lib/ourios/wal
  # RFC 0035: concurrent Parquet-encode workers (default: all cores).
  encode_workers: 4

querier:
  enabled: true
  http_addr: 0.0.0.0:4319
  default_window_secs: 3600
  mcp:
    enabled: false

auth:
  # See the Authentication guide. Omit the whole section for open
  # mode (development only — the server warns once at startup).
  tokens:
    - name: edge-collector
      token: ${env:OURIOS_EDGE_TOKEN}
      tenants: [checkout, payments]
  oidc:
    issuer: https://dex.example.com
    audience: ourios-collector
    tenant_claim: groups              # optional once openfga binds the tenants
    name_claim: name
    # RFC 0047: agent principals + group claims for the graph resolver
    agent_claim: ourios_principal_type=agent
    groups_claim: groups
  openfga:                            # RFC 0047 — the graph binds tenants
    api_url: http://openfga.auth.svc:8080
    store_id: 01M07RYMXRDW4ND5M7XQV04W8R
    authorization_model_id: 01M07RZE9RHPVPTYCV22RX0TDA   # pinned; omit = latest
    api_token: ${env:OURIOS_OPENFGA_TOKEN}                 # ${env:…} only
    session_ttl_secs: 60
    consistency: minimize_latency     # or higher_consistency
    request_timeout_secs: 5
    server_list_objects_deadline_ms: 3000
    visibility:                       # RFC 0047 §3.4 — layer 2 inside a tenant
      objects:
        - type: conversation
          column: attr.gen_ai.conversation.id
      # Optional (RFC 0048 §3.2). Which promoted columns carry the
      # principals in a conversation. An omitted list takes the semconv
      # default shown here and is exempt from the promoted-column check;
      # every EXPLICITLY listed entry must be a promoted column (startup
      # error otherwise). self_principal_column, when set, must be a
      # promoted column AND one of user_columns.
      # identities:
      #   user_columns: [attr.user.hash, attr.enduser.pseudo.id]
      #   agent_columns: [attr.gen_ai.agent.id]
      self_principal_column: attr.user.hash
      # Optional. REPLACES the default set (body + the GenAI content
      # attributes) — list every column to mask; must not be empty.
      # content_columns: [body, attr.gen_ai.input.messages, attr.gen_ai.output.messages]
      max_objects: 10000
      list_timeout_ms: 2000           # must be < server_list_objects_deadline_ms

Environment variables (no --config)

Auth configuration is file-only — there are deliberately no OURIOS_AUTH_* variables; token values reach the file through ${env:…} references.

There is no tenant-derivation configuration: the tenant is named out of band on every export (RFC 0046) — the X-Ourios-Tenant header over OTLP/HTTP, x-ourios-tenant metadata over OTLP/gRPC — exactly as the querier’s X-Ourios-Tenant. It is required in open mode too (no default tenant) and, with auth on, must be one of the credential’s tenants. A Collector sets it once, e.g. exporters.otlp.headers.x-ourios-tenant: acme (or per request via the headers_setter extension). Resource attributes such as service.name describe the producer and never choose the tenant.

Docker

The release pipeline publishes a multi-arch image (amd64 + arm64) to GHCR, cosign-signed keyless:

docker pull ghcr.io/jensholdgaard/ourios:0.1.1

Verify the signature before trusting it — the identity is pinned to the exact release tag, so substitute both occurrences of the version when verifying another release (SECURITY.md is the authoritative verification reference):

cosign verify \
  --certificate-identity 'https://github.com/jensholdgaard/ourios/.github/workflows/image.yml@refs/tags/v0.1.1' \
  --certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
  ghcr.io/jensholdgaard/ourios:0.1.1

Image variants

Every release publishes three signed multi-arch images from the same source:

• default (:<version>) — glibc binary on distroless/cc.
• -static (:<version>-static) — static musl binary on distroless/static: no libc, libgcc, or libssl in the image, so the OS-package vulnerability surface scanners report is ~empty. Pick this one for the strictest supply-chain posture with the operational niceties (CA bundle, tzdata, nonroot passwd entry) kept.
• -scratch (:<version>-scratch) — the same musl binary on bare scratch, plus only the CA bundle TLS needs. Nothing else in the filesystem: the absolute minimum attack surface, but also zero operational conveniences — no tzdata, no passwd entry, and CA-bundle updates arrive only with Ourios releases rather than base-image bumps.

All three run identically (same flags, ports, and config surface below).

Run

Same binary, same configuration surface as the quickstart — env vars, or a mounted config file:

docker run --rm \
  -p 4317:4317 -p 4318:4318 -p 4319:4319 \
  -v ourios-data:/var/lib/ourios \
  -e OURIOS_BUCKET_ROOT=/var/lib/ourios/data \
  -e OURIOS_WAL_ROOT=/var/lib/ourios/wal \
  -e OURIOS_RECEIVER_ENABLED=1 \
  -e OURIOS_QUERIER_ENABLED=1 \
  ghcr.io/jensholdgaard/ourios:0.1.1

With a config file instead (the production posture — auth lives in the file):

docker run --rm \
  -p 4317:4317 -p 4318:4318 -p 4319:4319 \
  -v ourios-data:/var/lib/ourios \
  -v "$PWD/ourios.yaml:/etc/ourios/ourios.yaml:ro" \
  -e OURIOS_EDGE_TOKEN \
  -e OURIOS_S3_ACCESS_KEY_ID -e OURIOS_S3_SECRET_ACCESS_KEY \
  ghcr.io/jensholdgaard/ourios:0.1.1 \
  --config /etc/ourios/ourios.yaml

Secrets stay out of the file via ${env:…} references — pass through every variable your file references (the example forwards the auth token and, for an S3-backend file like the Configuration example, the store credentials; a local-backend file needs neither OURIOS_S3_* variable). The server shuts down gracefully on SIGTERM — docker stop flushes the ingest pipeline before exit.

Local note: any OCI runtime works — with containerd, nerdctl run/nerdctl compose take the same arguments.

Kubernetes (Helm)

The chart at deploy/helm/ourios deploys the production topology: three workloads, one binary, backed by S3-compatible object storage (AWS S3, MinIO, R2, Ceph/RGW, … — RFC 0019):

• receiver — a StatefulSet with a per-replica WAL PVC (the WAL is local by design; only data + audit go to S3);
• querier — a stateless Deployment, scales independently;
• compactor — a singleton Deployment.

Install

kubectl create secret generic ourios-s3 \
  --from-literal=OURIOS_S3_ACCESS_KEY_ID=… \
  --from-literal=OURIOS_S3_SECRET_ACCESS_KEY=…

helm install ourios deploy/helm/ourios \
  --set storage.backend=s3 \
  --set storage.s3.bucket=ourios-logs \
  --set storage.s3.region=eu-central-1 \
  --set storage.s3.endpoint=https://s3.eu-central-1.amazonaws.com \
  --set storage.s3.existingSecret=ourios-s3

(On AWS EKS, IRSA replaces the secret — leave existingSecret empty and annotate the service account with the role ARN; the two modes are mutually exclusive.)

The chart renders an RFC 0020 config file into a ConfigMap; credentials reach it as ${env:…} references resolved from the secret — never inline.

The chart’s README is the authoritative reference: full values.yaml documentation, the topology diagram, local-development (MinIO) recipes, and sizing notes. This page stays a pointer so the two never drift.

Sending and querying

In-cluster, point Collectors at the receiver Service (ourios-receiver:4317) and query the querier Service on 4319 — fronted by whatever ingress/TLS termination your cluster standardises on. Configure authentication before exposing either beyond the cluster boundary.

Authentication

Three postures, one enforcement path (RFC 0026 + RFC 0029), plus an optional authorization graph behind them (RFC 0047). Whatever authenticates a request, the result is the same (name, read tenants, write tenants) binding: ingest batches must fall entirely inside the binding’s write set (whole-batch 403 otherwise, before the WAL), queries and MCP tool calls enforce the read set, and the name labels the audit trail and metrics. Static tokens and OIDC claims bind both sets identically; the graph binds them separately. Rejections are deliberately undifferentiated — one 401 shape, no probing oracle.

Naming the tenant (every posture)

Authentication decides which tenants a caller may touch; the request itself must still name the one it means (RFC 0046: tenancy is out-of-band — never derived from the OTLP payload). Every ingest export carries the tenant selector — the x-ourios-tenant gRPC metadata key, or the X-Ourios-Tenant HTTP header (HTTP header names are case-insensitive) — naming the tenant the whole export lands in; an export without it is rejected (gRPC INVALID_ARGUMENT / HTTP 400) before any WAL work. Queries name their tenant with the same HTTP header; MCP tool calls instead pass it as the tenant tool argument — the /mcp route does not read the header. With a Collector, set the selector on the exporter:

exporters:
  otlp:
    endpoint: ourios.example.com:4317
    headers:
      x-ourios-tenant: checkout

One export = one tenant: a pipeline feeding several tenants runs one exporter per tenant (Collector routing connectors compose cleanly with this). The named tenant must fall inside the caller’s write set, or the whole batch is rejected (gRPC PERMISSION_DENIED / HTTP 403).

Open mode (development only)

No auth section at all. Every request passes unbound; the server warns once at startup. Never expose an open-mode listener beyond localhost or a trusted network segment.

Static bearer tokens

The Collector-friendly baseline — static credentials in the config file, values injected via ${env:…} (inline literals fail startup):

auth:
  tokens:
    - name: edge-collector
      token: ${env:OURIOS_EDGE_TOKEN}
      tenants: [checkout, payments]   # or ["*"] for all tenants

Senders attach Authorization: Bearer <token>; with a Collector:

extensions:
  bearertokenauth:
    token: ${env:OURIOS_EDGE_TOKEN}
exporters:
  otlp:
    endpoint: ourios.example.com:4317   # TLS by default; gRPC host:port
    auth:
      authenticator: bearertokenauth

Comparison is constant-time; token values never appear in logs, errors, metrics, or audit events — only the name does.

OIDC (JWTs from an identity provider)

Adds standards-based machine identity in front of the same enforcement — any conforming issuer works; Dex (CNCF) is the recommended lightweight deployment and the one the acceptance suite runs against:

auth:
  oidc:
    issuer: https://dex.example.com
    audience: ourios-collector        # your client id
    tenant_claim: groups              # a string-list claim → the tenant set
    name_claim: name                  # the audit/metric label

Verification is local: the issuer is contacted once at startup (discovery + JWKS — an unreachable issuer fails startup, by design) and again only when an unseen key id appears (rotation). Signatures verify against the asymmetric allow-list only — RS256/384/512, PS256/384/512, ES256/384; alg: none and HMAC never verify.

Machine senders use the OAuth2 client-credentials flow — with a Collector this is zero custom code:

extensions:
  oauth2client:
    client_id: ourios-collector
    client_secret: ${env:DEX_CLIENT_SECRET}
    token_url: https://dex.example.com/token
    scopes: [openid, profile, groups]
exporters:
  otlp:
    endpoint: ourios.example.com:4317   # TLS by default; gRPC host:port
    auth:
      authenticator: oauth2client

Both halves coexist in one config — a static-token Collector and JWT-bearing senders authenticate side by side, each confined to its own tenant binding.

OpenFGA (relationship graph binds the tenants)

RFC 0047 adds OpenFGA as an authorization layer behind either authenticator — OpenFGA never authenticates. Once a bearer is known (static token or verified JWT), the principal is mapped (service_account:<token name>, user:<sub>, or agent:<sub> when the token carries the configured agent_claim) and the graph answers which tenants it may query (can_query) and write (can_write), using the in-tree model deploy/openfga/model.fga:

auth:
  tokens:
    - name: collector-cluster1
      token: ${env:OURIOS_COLLECTOR_TOKEN}
      tenants: ["*"]                  # the graph decides; a list here only narrows
  oidc:
    issuer: https://dex.example.com
    audience: ourios
    groups_claim: groups              # → contextual team#member tuples
    agent_claim: ourios_principal_type=agent
    # tenant_claim is optional here — the graph binds the tenants
  openfga:
    api_url: http://openfga.auth.svc:8080
    store_id: 01M07RYMXRDW4ND5M7XQV04W8R
    authorization_model_id: 01M07RZE9RHPVPTYCV22RX0TDA   # pinned; omit = latest
    api_token: ${env:OURIOS_OPENFGA_TOKEN}                 # ${env:…} only
    session_ttl_secs: 60              # revocation latency = binding cache TTL
    consistency: minimize_latency     # higher_consistency bypasses OpenFGA's cache

Grants are administrative tuples written through OpenFGA’s own API or CLI, never by Ourios: tenant:acme#reader@user:alice, tenant:acme#writer@service_account:collector-cluster1, tenant:acme#owner@team:platform#member. A token’s group claim rides along as request-scoped team:<group>#member@<principal> tuples (never persisted; at most 100), so team membership needs no sync pipeline. A credential’s own tenant list — a static token’s tenants, an OIDC tenant_claim — can only narrow what the graph grants, never widen it; a principal the graph grants nothing is unbound (401).

Visibility inside a tenant (layer 2)

With the graph configured, every query also runs the RFC 0047 §3.4 two-step for the principal in the tenant it queries — query rewrite at plan time, never per-record checks:

1. Check(principal, can_read_content, tenant) allowed → the tenant predicate only (today’s plan).
2. else Check(can_read_metadata, tenant) allowed → every row, with the configured content_columns returned as null (body as {"kind":"masked"}); a query that filters or aggregates on one of them is 403 column_forbidden, naming the column.
3. else the principal is scoped (bound through scoped_reader): its readable conversations are enumerated through the streamed ListObjects — filtered to this tenant, at most max_objects tenant ids, within list_timeout_ms — and become attr.gen_ai.conversation.id IN (…), OR’d with the self fast path (self_principal_column == <subject>, user: principals only). Past the bound: 403 visibility_bound (“ask for tenant-wide read”); a cut-off stream: 503 visibility_incomplete — never a partial predicate. Template-level queries (drift, list_templates, template_drift) need tenant-wide content read (403 visibility_scoped).

auth:
  openfga:
    # …
    server_list_objects_deadline_ms: 3000   # OPENFGA_LIST_OBJECTS_DEADLINE
    visibility:
      objects:
        - type: conversation
          column: attr.gen_ai.conversation.id   # a promoted column
      self_principal_column: attr.user.hash     # optional; user: principals only
      # content_columns: [...]                  # optional; REPLACES the default set
      max_objects: 10000                        # tenant ids only
      list_timeout_ms: 2000                     # MUST be < the server deadline

content_columns defaults to the GenAI content attributes plus body; an explicit list replaces that set (list every column to mask) and may not be empty — masking is never silently disabled. objects unset means scoped principals see nothing (their bound is not enumerable). Tenant-scoped graph objects are tenant:<T> and conversation:<enc(T)>/<id> where enc percent-encodes / and % in the tenant, so tenants containing / never alias; a tenant that cannot be an object id (:, #, whitespace) has no graph objects and every question about it fails closed. The two Checks cache with the session TTL; the enumeration runs per query. The branch a query took is recorded on ourios.query.visibility (ourios.query.visibility.branch) and the request span.

Feeding the graph from the data

With a conversation object bound, Ourios writes the data-derived tuples itself (RFC 0047 §3.3) — nobody hand-writes conversation grants: for every stored row the compaction sweep rewrites, and for every batch the receiver flushes, the emitter derives conversation:T/<id>#parent@tenant:T, #participant@user:<user.hash | enduser.pseudo.id> (plus tenant:T#scoped_reader@user:<…>), #actor@agent:<gen_ai.agent.id> (plus its binding tuple) and the per-tenant tool:T/<name>#parent@tenant:T objects, and writes them in idempotent ≤ 100-tuple batches (ourios.graph.tuples). Operators write only the administrative tuples (tenant#reader/writer/owner/ metadata_reader, team#member, tool#caller, conversation#delegate).

Erasing a conversation (RFC 0047 §3.6 / RFC 0048 §3.3): the front door is the CLI, run with the daemon’s config —

ourios-server --config ourios.yaml graph erase --tenant acme --conversation c-7
ourios-server --config ourios.yaml graph erasures            # pending + phase

erase writes the durable request marker (erasure/tenant_id=<enc>/conversation=<enc>, create-if-absent, so a repeat never resets an in-flight erasure) and the next sweep rewrites every partition of the tenant with the conversation’s rows dropped, then deletes its graph tuples, then writes a conversation_erased audit event, logs one ourios.compaction.erasure.completed event and removes the marker. Rows first, tuples after — a dangling tuple is harmless, a dangling row is a leak.

Backfilling history (RFC 0048 §3.4): data stored before the graph was configured is fed to it with

ourios-server --config ourios.yaml graph backfill --tenant acme [--from 2026-08-01T00:00:00Z]

— reads every partition of the tenant (--from keeps partitions whose UTC hour starts at or after it), derives and writes the same tuples the sweep would, in idempotent ≤ 100-tuple batches, and never rewrites Parquet. Resumable: run it again after an interruption. Backfill and erasure exclude each other through the store: backfill refuses while erasures are pending, and the sweep defers a tenant’s erasures while a backfill lock exists (graph backfill --unlock --tenant acme clears a crashed run’s lock; graph erasures lists both marker kinds).

Both verbs are instrumented like the daemon: each run is one OpenTelemetry CLI callee span (process.executable.name, process.pid, process.exit.code, and error.type on failure), the backfill’s per-partition progress events reach stderr, and everything exports over OTLP when the standard variables point somewhere — OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4317. Silence it the standard way: OTEL_SDK_DISABLED=true for everything, or per signal with OTEL_TRACES_EXPORTER=none, OTEL_METRICS_EXPORTER=none and OTEL_LOGS_EXPORTER=none.

The binding is cached per credential for session_ttl_secs and is fail-closed: an unreachable or slow OpenFGA answers 503 on the query and MCP surfaces and UNAVAILABLE/503 on ingest, and ourios.auth.resolutions counts the failure as error.type = upstream_unavailable. Static tokens and OIDC keep working without an openfga section; a deployment that wants coarse tenants only never touches it.

TLS

The listeners speak plaintext today; terminate TLS in front (ingress, service mesh, or an L4 proxy) — bearer tokens over plaintext are not auth. Native listener TLS is tracked on the auth epic.

Query DSL by example

Sixteen sample queries, each stated first in plain English and then in the logs DSL (RFC 0002), followed by a migration sketch from Grafana Loki’s LogQL and Amazon CloudWatch Logs Insights. This page is also the RFC 0002 §9 readability sheet: if a query’s meaning is not obvious from the English line above it, that is a defect in the DSL — file an issue.

Every query is a single line: a predicate (what rows match), optionally followed by pipe stages (window, aggregate, sort, limit, project, render). A bare true predicate means “no filter”. Queries without a range(...) stage get the server’s default look-back window.

The samples

1. Every error or worse in the last hour:

   severity >= error | range(-1h, now)
   

2. Everything the checkout service logged in the last 15 minutes:

   service == "checkout" | range(-15m, now)
   

3. Errors from checkout whose body mentions a timeout:

   service == "checkout" and severity >= error and contains(body, "timeout")
   

4. How many log lines each service produced today, busiest first:

   true | range(-24h, now) | count by service | sort count desc
   

5. Error count per template, to find the noisiest failure shape:

   severity >= error | count by template_id | sort count desc | limit 20
   

6. The ten most recent lines of one template, rendered back to the original bytes (reconstructed from the template, or the retained body verbatim for lossy rows):

   template_id == 42 | sort ts desc | limit 10 | render
   

7. Lines matching a regular expression (matched against the body; add ^/$ yourself if you want anchoring):

   matches(body, "user [0-9]+ locked out")
   

8. Everything with a given trace id, across all services:

   trace_id == "4bf92f3577b34da6a3ce929d0e0e4736"
   

9. Warnings and errors, excluding a known-noisy service:

   severity >= warn and service != "vacuum-daemon"
   

10. Lines carrying a specific attribute value (any OTLP attribute is addressable, promoted or not):

    attr.decision == "deny" | range(-6h, now)
    

11. Kubernetes-style resource lookup — one pod’s logs:

    resource["k8s.pod.name"] == "checkout-7d4b9f-x2m8p" | range(-30m, now)
    

12. Total LLM spend by model over the last day (typed numeric attribute, RFC 0042):

    true | range(-24h, now) | sum(attr.cost_usd) by attr.model
    

13. Login-failure count per five-minute bucket — a rate over time:

    template_id == 42 | range(-3h, now) | count by bucket(5m)
    

14. The same failures broken down by the template’s first parameter (e.g. which user), pinned to one template:

    template_id == 42 | range(-3h, now) | count by param(0)
    

15. Structured OTel events by name (RFC 0043/0044):

    event_name == "gen_ai.client.inference.operation.details" | limit 50
    

16. Low-confidence parses that kept their original body — the miner’s own honesty check:

    lossy == true and confidence < 0.5 | project ts, service, body
    

Migrating from LogQL (Grafana Loki)

The structural difference: Loki selects streams by label matchers in {braces}, then pipes line filters; Ourios has no stream/label split — everything (service, resource attributes, log attributes, severity, template) is one predicate namespace, and aggregation is a pipe stage instead of a wrapping function.

Two things have no LogQL counterpart and come free here: template_id (query the shape of a line, pre-clustered at ingest — no regex needed to group “user N logged in” lines) and param(n) (group by a template’s wildcard value without an extraction stage).

Not carried over from LogQL: unwrap-style duration/bytes conversions and metric-query arithmetic (/, + between range vectors) — the DSL returns counts and typed-attribute aggregates, and arithmetic between result sets is the caller’s job today.

Migrating from CloudWatch Logs Insights

Insights is closest in spirit — a pipe language over discovered fields — so most queries transliterate stage by stage:

The parse-then-stats idiom is the one to notice: what Insights does with an ad-hoc glob at query time, Ourios did once at ingest — the template is a stored column, so the grouping needs no pattern and costs no scan-time extraction.

Not carried over from Insights: cross-log-group joins and dedup — out of scope for v1 of the DSL.

Observe your coding agent

Coding agents (Claude Code, GitHub Copilot CLI, …) emit OpenTelemetry already. Point that telemetry at a local Ourios and you close a loop: the agent’s own API cost, token usage, and tool decisions land as OTLP logs, and the agent can query them back — through Ourios’s MCP surface — about itself.

The whole loop runs on your machine. Agent telemetry is sensitive (prompts, tool output, source, and whatever flows through them), so the value here is that none of it leaves the host — no SaaS, no phone-home. That is the point, not a footnote.

Three steps: run Ourios, point the agent at it, ask the agent about itself.

1. Run Ourios

Aggregating by attribute (count by attr.model, sum(attr.cost_usd)) needs those attributes promoted to columns, which is a config-file setting (RFC 0022). Write an ourios.yaml:

storage:
  backend: local
  local:
    bucket_root: /var/lib/ourios/store
  # Claude Code emits these as flat log-attribute keys (not the OTel
  # GenAI semantic-convention dotted names); promote the ones you want to
  # group or sum by.
  promoted_attributes:
    log: [model, cost_usd, tool_name, decision]
receiver:
  enabled: true
  grpc_addr: "0.0.0.0:4317"
  http_addr: "0.0.0.0:4318"
  wal_root: /var/lib/ourios/wal
querier:
  enabled: true
  http_addr: "0.0.0.0:4319"
  mcp:
    enabled: true

Run it — mapping the ports to loopback only (127.0.0.1:), because this config has no auth section and so runs open (RFC 0026); a bare -p 4318:4318 would expose an unauthenticated receiver to your LAN:

docker run --rm \
  -p 127.0.0.1:4317:4317 -p 127.0.0.1:4318:4318 -p 127.0.0.1:4319:4319 \
  -v ourios-data:/var/lib/ourios \
  -v "$PWD/ourios.yaml:/etc/ourios/ourios.yaml:ro" \
  ghcr.io/jensholdgaard/ourios:0.1.1 \
  --config /etc/ourios/ourios.yaml

See Docker for image variants and signature verification. docker stop flushes the ingest pipeline on SIGTERM.

2. Point the agent at it

Telemetry is read at process startup, so set these, then start a new agent session. The enable flag alone ships nothing — you also need the exporter block.

# where + how (any OTLP-log source)
export OTEL_LOGS_EXPORTER=otlp
export OTEL_METRICS_EXPORTER=none          # Ourios is logs-only
export OTEL_TRACES_EXPORTER=none
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
export OTEL_SERVICE_NAME=my-agent
export OTEL_EXPORTER_OTLP_HEADERS=x-ourios-tenant=my-agent   # → the Ourios tenant

# then the per-tool enable flag:
export CLAUDE_CODE_ENABLE_TELEMETRY=1      # Claude Code
# export COPILOT_OTEL_ENABLED=true         # Copilot CLI

The tenant is named out of band by the exporter header (OTEL_EXPORTER_OTLP_HEADERS=x-ourios-tenant=…, RFC 0046) — never by service.name, which stays a plain resource attribute you can filter on. The tenants that exist show up as directories under bucket_root/data/ — e.g. /var/lib/ourios/store/data/tenant_id=my-agent/.

Prompt and tool bodies are not captured by default. Turn them on only on data you’re willing to retain — this is the sensitive part:

export OTEL_LOG_USER_PROMPTS=1 OTEL_LOG_TOOL_DETAILS=1   # Claude Code, opt-in

3. Add the MCP surface, and ask

Connect the same agent to Ourios’s read-only MCP:

claude mcp add --transport http ourios http://127.0.0.1:4319/mcp

Now ask the agent about itself in plain language — it reads the ourios://query-schema resource and composes the query:

What has my-agent cost so far, by model? Use the ourios tools.

Queries are scoped by tenant — the x-ourios-tenant value the exporter header set (my-agent above), not a single run — so this reports every session that shipped under that tenant, not just the current one (see the last note below).

Or query the DSL directly. First find your templates — the list_templates MCP tool (or just ask the agent) lists each template_id with its rendered text; note the id of the claude_code.api_request template. Then:

# your tool-use distribution
curl -s http://127.0.0.1:4319/v1/query \
  -H 'X-Ourios-Tenant: my-agent' -H 'Content-Type: text/plain' \
  -d 'true | count by attr.tool_name'

# total spend per model — substitute the api_request template id for <ID>
curl -s http://127.0.0.1:4319/v1/query \
  -H 'X-Ourios-Tenant: my-agent' -H 'Content-Type: text/plain' \
  -d 'template_id == <ID> | sum(attr.cost_usd) by attr.model'

Things that will trip you up

• Query by template_id or an attribute, not severity. These events carry severity_number 0 (unset), so severity >= trace silently excludes them. Use template_id == N (find N with list_templates) or a promoted attribute, or the match-all true.
• Fresh records sit in the WAL for up to ~5 minutes before they flush to Parquet, and the querier reads Parquet only — so a query right after a burst of activity can come back empty. Give it a few minutes, or docker stop (which flushes) and restart.
• Promotion is write-side. It applies to telemetry captured after the server starts with the config above; earlier data has no attr.model column to group on.
• CLAUDE_CODE_ENABLE_TELEMETRY=1 alone exports nothing — the OTEL_LOGS_EXPORTER/endpoint block in step 2 is what ships the logs.
• Queries are tenant-wide, not per-session. The tenant is the exporter-header value, so every session that shipped under it aggregates together (service.name is producer metadata you can filter on, not the tenant). To scope to one run, promote session.id (add it to promoted_attributes.log) and filter on it — e.g. attr.session.id == "…" | sum(attr.cost_usd) by attr.model.

Where to next

• Authentication — before any listener leaves loopback, put a token or OIDC in front (the config above is open).
• RFC 0027 — the MCP surface; RFC 0032 — the query-schema resource the agent reads to compose queries.
• RFC 0002 — the full query DSL, including the count/sum/min/max/avg aggregations.

OTLP log format — what crosses the wire vs. what Ourios consumes today

Status: investigation finding. Drafted 2026-05-13 to answer “is our template miner targeting the actual OTLP shape, or a made-up one?” Conclusion: the latter. This doc surfaces the gap and lists the RFC patches it implies; it does not change code.

The Ourios glossary commits the project’s ingest contract to OTLP over gRPC and HTTP — “we do not invent our own format” (docs/glossary.md, entry OTLP). The template-miner RFC (docs/rfcs/0001-template-miner.md) does not carry through on that commitment: §6.1’s record schema has eight fields, none of which exist on the OTLP wire, and the ingest signature today is MinerCluster::ingest(tenant_id, raw: &str) — a flat text line, not a structured LogRecord. This document closes the loop.

The first audience for this finding is the maintainer; the second is the RFC 0001 amendment PR and the future RFC 0003 (OTLP receiver) it implies.

1. What OTLP actually carries

The wire-level definition lives in opentelemetry-proto/opentelemetry/proto/logs/v1/logs.proto and the spec at opentelemetry.io/docs/specs/otel/logs/data-model. The relevant message hierarchy is:

LogsData
└── ResourceLogs[]
    ├── resource: Resource           ← Resource.attributes carries service.name, host.*, etc.
    ├── schema_url: string
    └── scope_logs: ScopeLogs[]
        ├── scope: InstrumentationScope   ← name, version, attributes
        ├── schema_url: string
        └── log_records: LogRecord[]

A single LogRecord carries:

Plus, inherited from the parent containers: the Resource attributes (the unit of “where did this come from” — typically service.name, host.name, k8s.pod.uid, etc.) and the InstrumentationScope name/version (which library/module emitted this record).

AnyValue is a oneof of: string_value, bool_value, int_value, double_value, array_value (recursive), kvlist_value (recursive map of strings → AnyValue), and bytes_value. The spec is explicit about the structured case:

Body MUST support AnyValue to preserve the semantics of structured logs emitted by the applications.

So a real OTLP emitter is at liberty to send a LogRecord whose body is, for example, {"msg": "user logged in", "user_id": 42, "from_ip": "10.0.0.1"} as a kvlist_value — with the parameters already structured out, not embedded in a free-text string.

2. What Ourios consumes today

MinerCluster::ingest(tenant_id: &TenantId, raw: &str) -> u64 (in crates/ourios-miner/src/cluster.rs). The pipeline:

1. tokenize(raw) splits on Unicode whitespace (crates/ourios-miner/src/tokenize.rs).
2. mask(tokens) runs UUID / IPv4 / NUM rules over the resulting &str slice (crates/ourios-miner/src/mask.rs).
3. descend + leaf lookup attaches to or creates a template (crates/ourios-miner/src/tree.rs).

The Parquet record promised by RFC 0001 §6.1 carries:

tenant_id, template_id, template_version, params,
separators, body?, confidence, lossy_flag

That’s the entire data model. Zero fields from the OTLP wire are reflected in the record.

That was the RFC 0001 record. The schema RFC 0017/0018 arrived at does store the OTLP fields (severity, trace context, attributes, scope), and template_id sits beside them as a sibling column — which is the thing to be clear about: it is an Ourios-derived, tenant-local u64, not an OTLP field, and not portable between stores (RFC 0010 drift and RFC 0007 aliases exist because of exactly that). A template’s portable identity is its string — log.record.template in the emerging OpenTelemetry convention (RFC 0050).

3. The gaps

3.1 Severity is missing from the record (and from the template key)

severity_number is one of the most common operator query filters: “show me all ERROR-or-worse from service.name = api in the last hour.” Today the miner has no severity field.

Worse: the template key doesn’t include severity. A line emitted at INFO and the same line emitted at ERROR would currently collapse to one template_id. That’s a §3.1-class problem (“no silent template merges”) in disguise — two semantically distinct events sharing one id.

3.2 Timestamps are missing from the record

time_unix_nano and observed_time_unix_nano carry the data that the B1 thesis gate (“predicate-pushdown query latency on time/template/tenant filters”) explicitly measures. Without a time column we cannot run B1 at all.

Today there is no time field on the record. The Parquet writer PR (Phase 2 in docs/roadmap.md) cannot land without RFC 0001 §6.1 amending to add at least time_unix_nano.

3.3 Resource and scope are missing

Resource.attributes is OTLP’s “who sent this” partition key — in real deployments, service.name is the natural partition for template trees (it’s effectively the per-service template namespace). Today our tenant_id is operator-supplied and has no declared mapping from OTLP fields. We need to decide: tenant_id := resource.attributes["service.name"]? Or some configured mapping rule? RFC 0003 (OTLP receiver) is the place for this; RFC 0001 just needs to make resource_attributes a record column so the decision can land.

InstrumentationScope.name distinguishes the same body text emitted from different code paths in the same service. Likely also belongs in the template key — myapp.login and myapp.checkout emitting "request received" are different events.

3.4 Attributes carry the structured params we try to mine

In a structured-logging world, the values our mask() rules try to extract from text (NUMs, IPs, UUIDs) are typically already typed and separated by the SDK as Attributes. A modern emitter sends:

• body = "user logged in"
• attributes = {"user.id": 42, "client.address": "10.0.0.1"}

Not:

• body = "user 42 logged in from 10.0.0.1"
• attributes = {}

Our miner gets the second form and does work to reconstruct roughly what the first form already had. Worse, given the first form, we currently mine "user logged in" as a flat fixed template and lose the typed attribute values entirely — they’d never reach the Parquet record. The operator query “show me all logins from client.address = 10.0.0.1” returns nothing.

The implication for the miner is significant: the params slot on the record cannot be only “things mask() extracted from the body string.” It must also carry the OTLP attributes of the record — either as a sibling column (operator-queryable) or folded into the existing params shape (more complex).

3.5 Body is not always a string

AnyValue body. Today ingest(raw: &str) cannot accept a structured body at all. Three plausible paths:

1. Render-to-string at the receiver. Convert structured Body to a canonical JSON-ish string before handing to the miner. Loses the structure but preserves the existing miner shape. Risk: §3.3 (“bit-identical body reconstruction”) requires the rendered form to round-trip; canonicalising arbitrary AnyValue trees is non-trivial.
2. Treat structured Body as not-mineable. Store it verbatim in the body? column with lossy_flag = false (it’s an explicit structured value, not a lossy reconstruction); the miner emits a template_id of “structured body” and the query path knows to read body? directly. Simpler, gives up templating for those records.
3. Mine inner string fields. If body is a kvlist_value with a "msg" field, mine msg as the line. Pragmatic but ad-hoc; the field name is convention not spec.

Path (2) is the cleanest minimum; path (1) is the eventual ambition; path (3) is a configurable convenience layer. All three need an explicit spec decision.

3.6 Trace correlation is missing

trace_id, span_id, flags are how operators correlate logs to spans in the same trace. Real operators use this constantly. Today: no fields, no support. Add as record columns.

3.7 The ingest signature itself is wrong

ingest(tenant_id: &TenantId, raw: &str) cannot accept any of the above. The eventual signature is roughly:

#![allow(unused)]
fn main() {
fn ingest(&mut self, record: &OtlpLogRecord) -> u64
}

…where OtlpLogRecord is a struct that mirrors the OTLP wire shape (or borrows directly from a tonic-decoded protobuf message). This is a breaking change to the cluster’s public surface and is rightly the territory of RFC 0001’s amendment.

4. Implications

4.1 RFC 0001 §6.1 needs amendment

The minimum schema additions to make the record OTLP-faithful:

Plus an explicit decision on:

• Template key. Is the leaf identified by (masked_body_tokens) alone, or by some tuple of (severity_number, scope_name, masked_body_tokens)?
• body representation. AnyValue → what does the miner see? (Per §3.5 above.)
• tenant_id derivation. What OTLP field(s) define it?

4.2 RFC 0001 §6.2 (algorithm) needs a tokenize/mask amendment

tokenize + mask are designed for text. Once Body is AnyValue, the front of the pipeline branches: structured Body skips the tokenize/mask path entirely (or uses path (3) above on a configured field). The algorithm spec needs to acknowledge this fork.

4.3 RFC 0003 (OTLP receiver) becomes a prerequisite, not a follow-up

Today’s roadmap.md §5 lists the OTLP receiver as “first post-MVP shipping PR series.” That sequencing assumes the receiver is just the wire-decode-and-forward layer for an already-OTel-aligned record schema. With the gaps in §3 above, the receiver and the schema co-evolve: you cannot define the record without knowing what the receiver hands you, and you cannot define the receiver without knowing what the record expects. RFC 0003 should be drafted alongside the RFC 0001 amendment, not after it.

4.4 The Phase-3 corpus + bench need an OTLP-shaped corpus

The corpus runner (ourios-bench, Phase 3) cannot validly exercise the C2 thesis gate (template-count convergence) on flat-text input if the production input is OTLP. The corpus input must itself be OTLP-shaped — either a pre-recorded batch of LogsData protobuf, or a generator that emits realistic LogRecords including the structured-Body and attributes-bearing variants.

4.5 The current cluster’s behaviour is not fully wrong, just narrow

Plain-text traditional logs (Syslog, Log4j, slog with default text formatter) produce LogRecords with string Body and near-empty Attributes. The current miner handles those records correctly modulo the missing timestamp / severity / resource columns. So the current code is not throw-away; it’s the text arm of a fork that the OTLP-aware ingest will need.

5. Recommendation

Three follow-ups, in order:

1. Patch RFC 0001 §6.1 + §6.2 (a meta:-shaped change to the record schema and the algorithm spec). Land the new columns, the template-key decision, and the AnyValue handling fork. Do this first because the rest of the work depends on it.

2. Draft RFC 0003 — OTLP receiver. Cover (a) the wire-decode layer (tonic for gRPC, axum/hyper for HTTP/protobuf, against the official opentelemetry-proto crate); (b) the OtlpLogRecord → MinerCluster mapping; (c) the tenant_id derivation rule; (d) the WAL-before-ack sequencing under the new structured shape (§3.4); (e) build-vs-depend evaluation (tonic + hand-roll vs. embedding the rotel Rust collector vs. running the OTel Collector out-of-process and forwarding).

3. Patch the miner crates to consume the new record shape and route Body through the AnyValue fork. Update the roadmap to reflect that OTel-native ingest is no longer strictly post-MVP for the C2 gate’s validity.

The user-visible effect: the eventual benchmarks measure what an actual OTel deployment would experience, not a flat-text caricature of it. The thesis claim of “Parquet + template mining

• DataFusion is the right stack for OTel logs“ becomes testable in the form an operator would actually evaluate it.

6. References

• OTLP logs.proto: github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/logs/v1/logs.proto
• OTLP common.proto (AnyValue, KeyValue): github.com/open-telemetry/opentelemetry-proto/blob/main/opentelemetry/proto/common/v1/common.proto
• OpenTelemetry Logs Data Model spec: opentelemetry.io/docs/specs/otel/logs/data-model
• RFC 0001 §6.1 (current record schema): docs/rfcs/0001-template-miner.md
• Glossary entry OTLP (the load-bearing commitment): docs/glossary.md

Last updated: 2026-05-13.

Hazards

Referenced from CLAUDE.md §4 (“Before any change to the hot path, re-read docs/hazards.md”) and §10 (“When in doubt: 1. Read docs/hazards.md”). This document is the load-bearing reading for any hot-path reviewer. Each hazard names a specific failure mode, the mitigation we have committed to, the detection signal, and the rule for when a deviation is a tuning question vs. an architectural one.

How to use this document

• Before opening a PR that touches any subsystem named in a hazard section: re-read that section. The PR description must explicitly say which hazard it touches and how the change preserves the mitigation.
• In review: if a hazard is touched and not addressed in the PR description, that is a block, not a nit.
• In production: the named detection signals are the alerts that cannot be silenced without an RFC. They exist precisely so the failure mode is visible before it corrupts data.

Hazards map onto invariants in CLAUDE.md §3. Hazards describe what goes wrong; invariants describe what we promised. They are two faces of the same constraint.

H1 — Template miner correctness

Failure mode. The miner merges semantically-distinct templates because they share token structure. The canonical horror: user logged in <*> and user logged out <*> differ in one token; below a permissive threshold they merge into user logged <*> <*>. A query for the login event silently returns logout rows. The operator never knows.

Mitigation.

• Default similarity threshold ≥ 0.7 (strict).
• Lowering the threshold below 0.7 requires an RFC, not a config change.
• Three-zone confidence model: clean match (≥ threshold) / lossy match (floor ≤ x < threshold, retain body — reconstruction still succeeds, so lossy_flag is not set) / parse failure (< floor, retain body, increment counter). lossy_flag is reserved for the H7 case (genuine tokenizer / preprocessing failure where reconstruct(record) != ingested_bytes is possible); it is not a low-confidence signal. See docs/rfcs/0001-template-miner.md §6.6 for the precise definition.
• Every template-widening event is audited: the audit record names the old template, the new template, tenant, timestamp, and reason.

Detection. All metrics carry tenant_id; some carry service.

• merges_total counter: spike on stable input → service-version change or threshold drift.
• body_retention_ratio gauge: rising → input shifted or threshold is too tight.
• confidence_p01 histogram tail: collapsing → many matches are barely passing; threshold should be revisited.
• parse_failures_total: nonzero is genuine failure, not lossy.

Escalation. A spike on one tenant is a tuning question (masking rules, per-tenant threshold). A spike across many tenants on a stable corpus is a policy question — RFC.

See also. CLAUDE.md §3.1; docs/rfcs/0001-template-miner.md §§6.3–6.4; docs/benchmarks.md C2 (template count convergence), C3 (merge rate).

H2 — Parameter cardinality blowup

Failure mode. A params slot captures something it should not — an entire stack trace, a base64 payload, a request body, a megabyte JSON blob. Parquet’s dictionary encoding for that column collapses (every value distinct). File sizes explode. Query latency on that column degrades by orders of magnitude. The backend’s compression claim evaporates for that workload.

Mitigation.

• Per-parameter byte limit, default 256 B, ceiling 1 KiB — raising the ceiling requires an RFC.
• Overflow spills the original value into the body column; the params slot is replaced by a truncation marker (length + hash, no original payload).
• Counter increments on overflow.

Detection.

• params_overflow_ratio per service: alert when > 1 % of lines for any one service hit overflow.
• Parquet column-size variance: a column whose dictionary efficiency drops sharply between compactions usually means a new overflow pattern.

Escalation. Service-specific spike → masking rule that pre-redacts the offending field. Broad spike → revisit the limit (still ≤ 1 KiB). Anyone proposing > 1 KiB → RFC.

Structured bodies (RFC 0037, proposed mitigation — not yet implemented). The per-parameter byte limit above guards the string path only. A structured (non-string) body — a GenAI event’s gen_ai.input.messages array, any AnyValue kvlist/array — is retained whole as canonical JSON in the body column and is not capped: truncating it would violate the bit-identical-reconstruction invariant (CLAUDE.md §3.3). Crucially, this hazard’s failure mode — dictionary-encoding collapse — does not apply to body: the Parquet writer disables the dictionary on the body column by design (crates/ourios-parquet/src/writer.rs §3.6 — bodies are unbounded and high-entropy), so a large structured body has no dictionary to collapse. The residual risk is raw storage size, not dictionary collapse, so RFC 0037 proposes to guard it by observation, not truncation: a structured_body_bytes metric (dimensioned by service) plus a per-service alert to flag oversized emitters, with RFC 0036’s write-side row-group/file sizing bounding the on-disk footprint. The fix for an oversized structured body is at the emitter (redaction/truncation before export), not a store-side cap.

See also. CLAUDE.md §3.2, §3.3; RFC 0001 §6.5; RFC 0037; benchmarks C4.

H3 — WAL durability vs. latency

Failure mode. The ingester acknowledges an OTLP batch before the write is durably persisted. The ingester then crashes (process kill, host failure, container reschedule). The producer believes the data was accepted; we have lost data we promised to keep.

Mitigation.

• An ack is emitted only after fsync (or equivalent durability primitive) on the WAL.
• Batched fsync with an explicit operator-tunable knob: default flush every 100 ms or when the current segment fills, whichever first.
• Crash-recovery test is part of CI: SIGKILL the ingester mid-batch, restart, assert no acknowledged data is missing. Test runs on every PR; failure blocks merge.
• Replication, when added, is in addition to the WAL, not a replacement.

Detection.

• ingest_ack_latency_p99: rising trend usually means fsync is the bottleneck.
• wal_unflushed_bytes: bytes acked but not yet on durable storage — must always be bounded.
• CI crash-recovery test: any failure is critical, regardless of flake history.

Escalation. Fsync latency rising → tune batch size or move to faster storage. Ack-without-fsync ever observed in code review → P0 bug, hotfix path.

See also. CLAUDE.md §3.4; RFC 0008 (WAL design); benchmark D2 (compaction keeps up).

H4 — The small-file problem

Failure mode. WAL segments get rotated and flushed to Parquet too eagerly. The result is thousands of small files per tenant per day. Object-storage LIST calls dominate query planning time. Cold cache hits are murderous. Operators see “query took 12 s on 4 GB of logs” and lose faith in the backend.

Mitigation.

• Target row-group size 128 MB – 1 GB for ingest-side Parquet files. Compacted files deliberately rotate row groups at a smaller threshold (RFC 0036 §3.3, initially 32 MiB) — the pruning-granularity knob: within one compacted file a smaller row group buys tighter per-group min/max statistics (so a windowed query scans the groups that hold its answer, not the whole hour) at the cost of a few more footer entries. File economics are governed by the file band below, which compaction leaves untouched.
• Target file size 256 MB – 2 GB post-compaction.
• Background compaction job per tenant; cadence is a tunable.
• Compaction is required to keep the WAL backlog bounded under sustained ingest (D2).

Detection.

• File-size histogram per tenant: fewer than 5 % of files below 128 MiB at steady state.
• File count vs. data volume: file count must grow sub-linearly with bytes ingested.

Escalation. Skewed file-size distribution on a single tenant → compaction tuning. Sustained small-file emission across the cluster → ingest-scaling block, RFC.

See also. CLAUDE.md §4 hazard 4; benchmarks D3.

H5 — Template schema evolution across deploys

Failure mode. A service ships a new version. Log format changes — a new field, a renamed token, reordered words. The template tree built from last month’s logs no longer matches the new format cleanly. Queries against template_id = X start returning incomplete results because some rows are now stored under template_id = X'. The operator sees a 30 % drop in event volume and misdiagnoses it as an outage.

Mitigation.

• Templates are versioned: a template’s internal representation can change; the logical identity persists across versions.
• Explicit alias mechanism: template_id.resolves_to(X) in the DSL resolves a query across all aliases of X.
• Drift detection is a first-class query — operators can ask “what templates drifted in the last 24 h?” and get a list.
• A new template_version emits an audit event, just like a merge.

Detection.

• Spike in distinct template count immediately after a deploy → expected; investigate only if it persists past the deploy window.
• Diff between template_id = X and template_id.resolves_to(X) result counts → measures alias coverage.
• Audit event volume: drift events should correlate with deploy cadence, not appear randomly.

Escalation. Alias graph becomes a tangle (templates with > N aliases or cycles) → revisit alias semantics, RFC. Drift correlated with deploys → expected; not an alert.

See also. CLAUDE.md §3.5; RFC 0001 §6.7.

H6 — Query DSL vs. DataFusion SQL surface

Failure mode. A user-facing query surface accidentally exposes DataFusion specifics — a SQL keyword leaks into an error message, a planner hint becomes documented, a join type that doesn’t make sense in a logs context becomes reachable. We then cannot upgrade DataFusion or change the planner without breaking saved user queries and dashboards. The DSL has become a contract we never intended to sign.

Mitigation.

• The DSL is a separately specified layer (docs/rfcs/0002).
• All DSL constructs compile to DataFusion LogicalPlan, never to SQL strings. SQL never appears in any user-visible output.
• No SQL escape hatch by default. If one is added later, it ships under a separate RFC, sandboxed, opt-in, and tenant-gated.
• DSL evolution is a written semver contract with users; major versions ship with a deprecation window.

Detection.

• PR review: any test or error message containing the substring “DataFusion” or referring to a DataFusion type by name in a user-facing surface is a block.
• Any code path that constructs SQL strings from user input is a block.
• User report: “this query worked yesterday after the upgrade” triggers a regression review.

Escalation. Leak found in user-facing surface → block + hotfix. Recurring temptation in implementation → tighten the API boundary, move shared helpers behind a non-exported module.

See also. CLAUDE.md §4 hazard 6; RFC 0002.

H7 — Bit-identical body reconstruction

Failure mode. An operator opens the UI and asks “show me what was actually logged.” We render the row from template + params and produce a string that drops a space, a quote, a separator, or a trailing newline. The operator chases a bug that doesn’t exist — or, worse, fails to chase a bug that does, because the rendered line looked normal.

Mitigation.

• The miner either captures inter-token whitespace and separators or it sets lossy_flag = true on the row. There is no third option.
• Reconstruction is a property test against the testdata corpus: for every non-lossy row, reconstruct(record) == ingested_bytes exactly. Property failure blocks merge.
• The reader honours lossy_flag. The UI surfaces lossy rows with an explicit warning (“this row’s body cannot be exactly reconstructed”) rather than rendering them.
• Tenants may opt into default-on body retention at a storage cost.

Detection.

• Reconstruction property test (CI): zero failures, ever, on the committed corpus.
• body_retention_ratio gauge: a sudden rise indicates input distribution change OR a regression in whitespace capture.
• User complaint of “the rendered log does not match what we sent” → reproduce, add to corpus, fix.

Escalation. Ever fails on a real-world corpus → block + hotfix. Whitespace-capture state machine becomes a complexity sink → simplify by retaining more bodies; the storage cost is real but acceptable, lying to the user is not.

See also. CLAUDE.md §3.3; RFC 0001 §6.6; benchmarks C1.

H8 — Replication-induced dedup under clock drift

Forward-looking hazard. Ourios does not currently replicate at ingest, so this hazard is dormant. It is recorded here so that any future RFC proposing replication starts with the failure mode already understood.

Failure mode. A multi-ingester replication design quietly introduces a storage multiplier if dedup is keyed on filename and a time window. Even sub-second clock drift between replicas causes the dedup pass to miss duplicates: each replica writes “the same” record under a slightly different filename or window, the dedup miss is invisible to any single replica’s logs, and the user pays for redundancy they thought they had bought once. A widely-deployed log backend was found in 2026 to be carrying a ~2.3× storage multiplier from exactly this failure mode, motivating a full re-architecture of its durability layer.

Mitigation.

• Currently: not a concern. Ourios does not replicate at ingest. Per CLAUDE.md §3.4, durability is per-ingester fsync on the WAL; per §3.6, object storage is the long-term truth. Replication, if introduced later, is “in addition to the WAL, not instead of it.”
• When replication is proposed: dedup MUST be content-keyed, never time-windowed. The producer (or the OTLP layer) supplies an idempotency key — a content hash combined with a producer identifier and a sequence number. Ingesters treat the key as opaque. Clock drift becomes irrelevant to dedup correctness.
• Time-window dedup is rejected by default, regardless of how cheap or convenient it appears. An RFC proposing it must read this hazard and address it explicitly.

Detection.

• bytes_stored / bytes_received per tenant: must hover near 1 on a single-replica deployment. A multiplier > 1 + ε on a single-replica deployment indicates double-writing somewhere upstream.
• On a future replicated deployment: this ratio should remain near 1 after dedup; a sudden rise indicates clocks drifted and the dedup pass is missing duplicates.
• A “dedup hit rate” metric on the dedup pass — a sudden drop signals that something is masking what should be visible duplicates.

Escalation. Replication proposed → RFC must explicitly address this hazard. Time-window dedup proposed → block, redirect to a content-keyed approach. Storage multiplier > 1.05 on a single- replica deployment → P0 investigation, something is double-writing.

See also. CLAUDE.md §3.4 (WAL durability), §3.6 (object storage as truth). Cautionary tale: Grafana Loki’s 2026 re-architecture replacing replicate-at-ingest with Kafka-as-durability (InfoQ news, April 2026).

Adding a new hazard

A new hazard belongs in this document if all of the following hold:

• It is a failure mode that silently corrupts data, lies to the user, or destroys the project’s value proposition.
• It is not obvious from reading the code (otherwise it is a bug, not a hazard).
• It has at least one named mitigation in the codebase or a committed RFC.

A new hazard is added via a meta: RFC, the same path as changes to CLAUDE.md §3 invariants.

Verification

Status: active. This document is the process spec. The proposed amendments to docs/rfcs/README.md and CLAUDE.md at the bottom of this file are tracked separately and applied in their own PR once the structure here is settled.

What this doc is for

The Ourios docs already define invariants (CLAUDE.md §3), hazards (docs/hazards.md), per-RFC testing strategies (RFC §6 — see §2.5), thesis-gates (docs/benchmarks.md), and the project’s testing discipline (CLAUDE.md §6.2). What is missing is the described process that connects them: how a contributor (human or agent) takes a §3 invariant or an H-x hazard, turns it into reviewable acceptance criteria, turns those into red tests, drives them green, and gates an RFC’s transition to accepted.

This doc fills that gap. It is human-readable process, not test code, not tooling, not a coverage policy.

1. The flow

Six links, four gates between them. The diagram below names them; the text after walks the chain in the order a contributor encounters it.

Invariant (§3)        Hazard (H-x)        RFC (§§1–4)
       \                 |                   /
        \________________|__________________/
                         ↓
                Acceptance criteria
              (RFC §5 — normative,
               structured prose)
                         ↓
                   Red tests
                  (compile, fail)
                         ↓
                  Green tests
              (unit + property + corpus)
                         ↓
                  Validated
        (corpus + thesis-gates pass on
         representative inputs)

The chain has two entry points (Invariant, Hazard) that converge on the third (RFC). An RFC enumerates the invariants and hazards it touches in its §1 Summary; reviewers verify the enumeration is exhaustive at the Drafted → Specified gate.

Invariant → RFC. A CLAUDE.md §3 invariant is a project-level promise. Until an RFC operationalises it, the invariant is a known debt. §4 Entry points describes the three doors into this chain.

Hazard → RFC. Each hazards.md H-x item names the RFCs and crates responsible in its Mitigation and See also fields. The hazard does not move; the RFC inherits the obligation to defend it.

RFC → Acceptance criteria. Acceptance criteria live in RFC §5 (see §2) and translate the invariants and hazards the RFC touches into testable scenarios. The Specified gate ratifies the list.

Acceptance criteria → Red tests. A red test is a compiling stub that fails — typically with todo!() or unimplemented!() — and references the scenario id in a doc comment. Red tests are not required at the Specified gate; they are the artefact of crossing the Red gate, immediately before implementation begins. Forcing stubs to compile at Specified would push authors into premature specificity about types and signatures. Red stubs are tagged #[ignore] so the outer CI loop stays green while the inner loop (the implementor running cargo test -- --ignored locally) sees the todo!()s fire as a TODO list — see §3 for the two-loop spec.

Red tests → Green tests. Implementation lands; each stub becomes a real test that passes; unit, property, and corpus tests cover the scenario as CLAUDE.md §6.2 dictates. The Green gate confirms every §5 acceptance criterion has a matching passing test.

Green tests → Validated. The thesis-gates in benchmarks.md §7 that the RFC’s pillars touch must pass on representative corpora. Once they do, the RFC’s status: flips to validated. Maintainer sign-off then flips it to accepted.

2. Acceptance criteria

The contract here is single-typed: every invariant or hazard the RFC touches resolves to one or more scenarios, each with an id, a leading clause grammar, and a greppable counterpart in test code.

2.1 Format

Structured prose using bold leading clauses. Each scenario carries a short numeric id (see §2.2) and follows the Given / When / Then / And pattern:

Scenario H1.1 — Semantically distinct templates do not silently merge

• Given a corpus containing user logged in <*> and user logged out <*>
• When similarity threshold is 0.7 (default)
• Then the two remain distinct template_ids
• And any widening produces an audit event recording both old and new templates

The format is the markdown the project already uses, not Gherkin. We do not adopt .feature files, cucumber-rs, or any other BDD tooling: the test code is Rust (CLAUDE.md §6.2), and the scenario lives in the RFC where reviewers are already reading. A second source of truth — a .feature file checked separately — would drift, and the tooling does not pay for itself at our scale.

2.2 Scenario ids

Three id grammars, chosen to make the source of the obligation visible at a glance:

• H<n>.<m> — hazard-rooted; H1.1 is the first scenario defending hazard H1.
• §3.<n>.<m> — invariant-rooted; §3.4.2 is the second scenario defending CLAUDE.md §3.4 (WAL-before-ack).
• RFC<NNNN>.<m> — RFC-internal; reserved for scenarios that defend an RFC’s own design decisions, not a numbered invariant or hazard. Example: RFC0001.3 for a Drain3-extension behaviour that is not load-bearing for any §3 invariant but is part of the RFC’s contract.

Numbers within an id family are assigned in the order scenarios are written and never renumbered. A retired scenario keeps its number; new scenarios append. This gives git log -S "H1.1" a stable target across the lifetime of the project.

2.3 Greppability

The id is referenced from the test code in a doc comment, exactly:

#![allow(unused)]
fn main() {
/// Scenario H1.1 — Semantically distinct templates do not silently merge.
/// See `docs/rfcs/0001-template-miner.md` §5.
#[test]
fn login_and_logout_do_not_merge_at_default_threshold() { /* … */ }
}

grep -R "H1.1" . then yields the scenario in the RFC, the test in the crate, and any cross-references in the docs — bidirectional in one command. If a scenario is renamed, both ends move in the same commit.

2.4 Normative vs. exhaustive

Acceptance criteria are the normative tests an RFC promises will exist. The implementation will write many more — regression tests, edge cases, performance smokes — and those are not catalogued in the RFC. Reviewers ratify the normative set: every invariant and hazard the RFC touches has at least one scenario, and the scenarios as written are testable in principle.

The opposite mistake — listing every test the implementation will ever write — turns the RFC into a test plan and freezes the implementation. We do not do that.

2.5 Location in the RFC

Acceptance criteria are a new RFC §5, immediately before Testing strategy. The placement is deliberate: criteria are the spec the testing strategy operationalises, so reviewers reading the RFC top to bottom encounter the what before the how. The proposed amendment to docs/rfcs/README.md at the bottom of this file captures the renumbering: existing §5 Testing strategy shifts to §6, Open questions to §7, References to §8.

3. The RFC maturity model

Five stages, four gates. Each stage is a value of the RFC’s status: frontmatter field, so an RFC’s current maturity is visible without reading the body:

accepted is a distinct terminal status — it represents maintainer sign-off after Validated is reached. rejected and superseded are the other terminals, all three reachable from anywhere in the maturity ladder. A Drafted or Specified RFC may be rejected on review without ever reaching Red; an Accepted RFC may be superseded by a later one without re-traversing the chain.

The table is the spec; the paragraphs below explain what artefacts exist at each stage and what a reviewer is ratifying.

Drafted. The RFC has §§1–4 (Summary, Motivation, Proposed design, Alternatives considered) plus §§7–8 (Open questions, References) filled enough that two engineers reading it would produce roughly the same implementation. Acceptance criteria (§5) and Testing strategy (§6) may be empty or stubbed. The PR is open with status: drafted; review focuses on whether the design is correct in principle. The gate to Specified is a peer reviewer saying “yes, this design is what we want — now write down the contract.”

Specified. §5 Acceptance criteria is filled. Every invariant in CLAUDE.md §3 and every hazard in hazards.md that the RFC touches has at least one numbered scenario. §6 Testing strategy references those scenarios and names the technique (proptest, corpus, criterion) for each. The reviewer asks one question: could a competent implementor turn each criterion into a test as written? If the answer is no — for any criterion — the RFC has a gap and goes back to Drafted.

The Specified gate is the most valuable. It is the only gate where the cost of being wrong is bounded by review time rather than implementation time. We do not require test stubs to compile here; forcing stubs would push authors into premature decisions about function signatures, traits, and module structure, which is the Red gate’s job, not this one.

Red. Test stubs exist, are tagged #[ignore], and fail when run. Each stub carries a doc comment naming its scenario id (§2.3). Stubs may be todo!(), unimplemented!(), assert!(false) — anything that compiles and fails. Implementation may begin.

The Red signal lives at two granularities, deliberately:

• Inner loop (local dev cycle). The implementor working on a stub runs cargo test <name> -- --ignored and watches the todo!() panic. Each panic is one TODO item; as the body fills in, the #[ignore] comes off and the test joins the default run.
• Outer loop (CI). Default cargo test skips ignored tests, so the Red-stage PR lands cleanly through branch protection rather than fighting it. CI’s signal that the Red gate is satisfied is structural: stubs compile, every §5 scenario has an #[ignore]’d test with a matching id, and cargo test -- --include-ignored exits non-zero on each. (The greppability contract in §2.3 makes the per-scenario coverage check mechanical — grep -R "H1.1" returning both the RFC line and the test stub line is the assertion.)

The two-loop split is what lets us treat the Red status as a landable, mergeable state rather than a half-broken branch. A Red-stage main is healthy: outer loop green, inner loop fully populated with the work that needs doing.

The gate is mechanical: every scenario in §5 has at least one stub with a matching id, the stub is tagged #[ignore], and cargo test -- --include-ignored exits non-zero on each.

Green. Implementation lands. Every stub becomes a real test; unit, property, and corpus tests cover their scenarios as CLAUDE.md §6.2 dictates. cargo test --all-features passes. The reviewer confirms each §5 criterion now resolves to a passing test (the greppability contract makes this mechanical). No performance claim is made yet.

Validated. Every thesis-gate in benchmarks.md §7 that the RFC’s pillars touch passes on representative corpora. Maintainer inspects the corpus and the delta against target, signs off, and flips status: to accepted. The RFC is now binding; subsequent changes go through the regression handling in §3.1.

3.1 Regression handling after Validated

A failing test on a previously-Validated RFC is, by default, the test doing its job. The RFC does not reopen. Standard PR workflow: fix the regression, ship the patch, the test stays green.

The RFC reopens only when a single criterion fails repeatedly on the same code path — concretely, when the same scenario id fails on three independent commits within a 30-day rolling window, or when two distinct regressions touch the same criterion within the same window. The threshold is that the criterion has stopped being a defence and has become a moving target; that is a signal the RFC’s commitment is under-defended or under-specified, and the design (not just the implementation) needs revisiting.

This threshold is informal at the Specified gate; it sharpens once real signals exist. The point of writing it down now is that contributors do not race to reopen RFCs on every CI flake, nor pretend a repeated structural failure is just bad luck.

Thesis-gate failures during Validated follow benchmarks.md §7’s existing escalation rule (one fail on one corpus → tuning RFC; two or more → pillar RFC, pause), not this section.

3.2 Outer loop vs. inner loop

The maturity model is the outer loop. Each stage names a checkpoint that an external reviewer can verify: at Specified the scenarios are written, at Red the stubs compile and fail, at Green the same stubs pass. Nothing in the outer loop says how a developer fills the Red → Green transition.

The recommended inner loop is classic Beck-style TDD: write one failing test, make it pass with minimal code, refactor, triangulate by writing the next test that forces generalisation, repeat. It is not mandatory — a developer who prefers to stub all scenarios up front and implement against them is welcome to. The outer loop only requires that every §5 scenario has a stub by the Red gate and a passing test by the Green gate.

Two consequences worth being explicit about:

1. More tests than scenarios. The inner loop typically writes many tests per scenario — one per concrete example, then regression tests as bugs surface. Acceptance criteria (§2.4) are the normative set the RFC is held to; the inner loop fills out the rest.
2. No refactor stage in the model. Refactoring is part of the inner loop, not a maturity stage. A Green or Validated RFC may be refactored without re-traversing the chain, as long as every §5 criterion stays green.

The split is the BDD/ATDD outer-shell convention adapted to a project already committed to Rust, proptest, and criterion: the scenarios are written in the BDD-flavoured prose of §2.1 because they live in RFCs and humans read them; the tests are written in the TDD-flavoured loop developers already know.

4. Entry points

The same machinery, three doors:

• Invariant entry — an item in CLAUDE.md §3. The criteria live in the RFC that operationalises that invariant; if no RFC yet exists for the relevant subsystem, the invariant is a known debt and the next RFC for that subsystem must address it.
• Hazard entry — an item in hazards.md. Each hazard’s Mitigation section names the RFCs and crates responsible; their acceptance criteria must reference the hazard id.
• RFC entry — a new RFC under docs/rfcs/. The RFC enumerates the invariants and hazards it touches in its §1 Summary; criteria in §5 must cover each.

5. Relationship to benchmarks.md

Correctness gates live here. Thesis-gates live in benchmarks.md §7. An RFC reaches Validated only when both:

• Every §5 acceptance criterion has a passing test, and
• Every thesis-gate in benchmarks.md §7 that the RFC’s pillars touch passes on representative corpora.

Single sentence; intentional non-duplication. benchmarks.md stays the performance owner.

6. Worked example

A concrete trace of the chain in §1, against an artefact that already exists. RFC 0001 Template miner is currently status: draft (becoming drafted once the amendment to docs/rfcs/README.md lands). Its operationalisation of CLAUDE.md §3.1 No silent template merges and hazards.md H1 Template miner correctness is the first place this process gets to bite on real material.

6.1 Invariant → RFC

CLAUDE.md §3.1 promises:

A template merge that crosses semantic boundaries (e.g. merging “user logged in” with “user logged out” because they share token structure) corrupts the backend.

hazards.md H1 names the canonical horror — user logged in <*> and user logged out <*> differing in one token, merging under a permissive threshold to user logged <*> <*>, a query for the login event silently returning logout rows.

RFC 0001 §6.4 Merge policy is the section that defends the invariant. As of the Drafted gate it commits to “When two templates become candidates for merge”, an audit event schema, and the rule “Default: strict. Never silent. No exceptions.”

6.2 RFC → Acceptance criteria

The Specified gate adds a new §5 to RFC 0001:

Scenario H1.1 — Semantically distinct templates do not silently merge

• Given a corpus containing user logged in <*> and user logged out <*>
• When similarity threshold is 0.7 (default)
• Then the two remain distinct template_ids
• And any widening produces an audit event recording both old and new templates

Scenario H1.2 — Lossy-zone match retains body

• Given a line whose best match has confidence in the lossy zone (floor ≤ x < threshold)
• When the line is ingested
• Then the body column contains the original line bytes
• And the row carries lossy_flag = false (the flag is reserved for tokenizer / preprocessing failure per docs/rfcs/0001-template-miner.md §6.6 — the lossy zone retains the body but reconstruction still succeeds)

Scenario H1.3 — Every widening emits an audit event

• Given any sequence of inputs that triggers a template widening
• When the widening completes
• Then an audit event exists naming the old template, the new template, the tenant id, the timestamp, and the event_type

Three scenarios cover §3.1’s three rules: do not merge across semantics, retain bodies on low confidence, audit every widening. Reviewers ratify that this is exhaustive against CLAUDE.md §3.1 and H1; they do not catalogue every edge-case test the implementation will write.

6.3 Acceptance criteria → Red tests

The Red gate adds three stubs to crates/ourios-miner/tests/:

#![allow(unused)]
fn main() {
/// Scenario H1.1 — Semantically distinct templates do not silently merge.
/// See `docs/rfcs/0001-template-miner.md` §5.
#[test]
#[ignore = "RFC 0001 Red gate — implementation pending"]
fn h1_1_login_and_logout_remain_distinct_at_default_threshold() {
    todo!("RFC 0001 §6.4");
}

/// Scenario H1.2 — Lossy-zone match retains body.
/// See `docs/rfcs/0001-template-miner.md` §5.
#[test]
#[ignore = "RFC 0001 Red gate — implementation pending"]
fn h1_2_lossy_zone_match_retains_body() {
    todo!("RFC 0001 §6.6");
}

/// Scenario H1.3 — Every widening emits an audit event.
/// See `docs/rfcs/0001-template-miner.md` §5.
#[test]
#[ignore = "RFC 0001 Red gate — implementation pending"]
fn h1_3_every_widening_emits_an_audit_event() {
    todo!("RFC 0001 §6.4");
}
}

Default cargo test skips the ignored stubs and passes (outer loop / CI green); cargo test -- --ignored exits non-zero with all three failing (inner loop / Red signal). The gate is satisfied; implementation may begin.

6.4 Red → Green

Implementation lands across ourios-miner (and supporting types in ourios-core). The three stubs become real tests: H1.1 ingests the two-template corpus, asserts two distinct template_ids, and queries the audit log for absence of widening events. H1.2 ingests a line whose token similarity falls in the lossy zone and asserts that the row’s body carries the original bytes and lossy_flag is false (the flag is reserved for the H7 reconstruction-failure case; see RFC 0001 §6.6). H1.3 ingests a sequence that provokes a widening and asserts the audit event’s structure.

cargo test --all-features passes. Reviewers confirm each H1.x id now resolves to a passing test via grep. No benchmark claim is made.

6.5 Green → Validated

benchmarks.md C2 Template count convergence is the thesis-gate that H1 most directly touches: if the miner is silently merging across semantics, template count grows wrong. The benchmark harness runs C2 on the LogPAI corpora and any self-collected corpus available, plots template count vs. lines ingested, and asserts the convergence target.

Once C2 passes — and any other thesis-gate the RFC’s pillars touch — the maintainer signs off. RFC 0001’s status: flips to accepted. The miner’s contract is now binding.

6.6 The failure mode that re-opens the RFC

A hypothetical: six months in, three independent PRs land that each add a workaround to keep H1.1 green — a special-case for common verb pairs, then for HTTP method tokens, then for log-level tokens. Each workaround is small, each test stays green. By the fourth PR, a reviewer notices: the criterion has stopped being a defence and has become a moving target. Per §3.1, the RFC reopens. The right answer is not a fifth workaround; it is to revisit RFC 0001 §6.4 — the merge policy itself is under-specified for the workloads we are seeing.

This is what the threshold in §3.1 is for. It is not a CI-flake counter; it is a signal that the design’s defence has eroded and needs to be redrawn before more code is written on top of it.

7. What this doc is not

• Not test-tooling guidance — proptest, criterion, etc. live in CLAUDE.md §6.2.
• Not a coverage policy — Ourios is a correctness project; line coverage is the wrong metric.
• Not an agent-instruction file — agents follow it because it is written down, not because it speaks to them.

8. Resolved decisions

Three questions raised during the outline review, decided before expansion so the rationale is preserved:

• Maturity stages appear in RFC frontmatter as the status: field. Reviewers and tooling see an RFC’s current stage without reading the body. See §3.
• Single regressions do not reopen a Validated RFC. A failing test on an existing criterion is the test doing its job; standard PR workflow applies. Repeated regression on the same criterion (rough threshold: same scenario id failing on three independent commits, or two distinct regressions touching the same criterion, both measured in a 30-day rolling window) signals the criterion has stopped being a defence and reopens the RFC. See §3.1.
• Thesis-gate failures during Validated follow benchmarks.md §7, not this doc. One thesis-gate failing on one corpus → tuning RFC; two or more → pillar RFC and an implementation pause.

Proposed amendment — docs/rfcs/README.md

Two changes. Shown as the new text:

In Required frontmatter

Update the status field’s valid values from the current four-state list to the five-stage maturity model plus terminals:

status: drafted | specified | red | green | validated | accepted | rejected | superseded

The maturity stages (drafted through validated) are gates an RFC moves through; accepted is the terminal post-maintainer-signoff binding state; rejected and superseded are the off-ramps. See docs/verification.md §3.

In Required sections

Insert a new item between the current §4 Alternatives considered and §5 Testing strategy, renumbering subsequent items:

5. Acceptance criteria — normative scenarios, one per invariant or hazard the RFC touches. Format: structured prose with Given / When / Then / And leading clauses; each scenario carries an id of the form H1.1, §3.4.2, or RFC<NNNN>.<m>, referenced from the test code so the mapping is greppable. See docs/verification.md §2.

Testing strategy shifts to §6, Open questions to §7, and References to §8.

In Lifecycle

Replace the current four-status list with the five-stage maturity model:

1. Drafted — PR opened with status drafted. Sections §§1–4 and §§7–8 are filled. Discussion happens in PR review.
2. Specified — §5 acceptance criteria are written, every invariant and hazard the RFC touches has at least one scenario, and review has confirmed the criteria are testable in principle.
3. Red — test stubs exist and fail. Implementation may begin.
4. Green — all acceptance criteria pass; unit + property + corpus tests green.
5. Validated — thesis-gates in docs/benchmarks.md §7 pass on representative corpora. Maintainer flips status to accepted.

A regression detected after Validated either reopens the RFC (if a criterion is invalidated) or spawns a tuning RFC per benchmarks.md §7 (if a thesis-gate degrades). See docs/verification.md §3.

The earlier superseded and rejected entries remain unchanged.

Existing RFC frontmatter

RFC 0001 and RFC 0002 currently carry status: draft. The amendment PR renames both to status: drafted so the maturity model applies uniformly. No content change to the RFCs themselves at that step.

Proposed amendment — CLAUDE.md

A single new subsection under §5 Development workflow, following §5.5 One-word mode:

5.6 Verification process

The path from invariant or hazard to passing test is described in docs/verification.md. Acceptance criteria live in RFC §5; docs/rfcs/README.md defines the maturity stages an RFC moves through. The shortest version of the rule: if a criterion cannot be turned into a test, the RFC has a gap.

No change to §6.2 Testing discipline; verification.md links to it. The §6.2 content (proptest, corpus tests, crash recovery, criterion) is the catalogue of techniques; verification.md is the process that decides which technique is required where.

Applying the amendments

The body of this document is the verification process spec. The two proposed amendments above are pending application:

• docs/rfcs/README.md — status: value list, new §5 Acceptance criteria in Required sections with renumbering, lifecycle rewrite, draft → drafted rename in RFC 0001 and 0002.
• CLAUDE.md — new §5.6 Verification process.

Both should land in a single PR. RFC 0001 then gets a §5 Acceptance criteria applied as the first concrete use of the process — the worked example in §6 of this document is the target shape, and applying it will probably surface specificity gaps in RFC 0001’s existing design. That surfacing is the point.

Add this document to docs/SUMMARY.md under the Architecture header in the same PR that applies the amendments.

Glossary

Vocabulary used in the Ourios docs. Entries marked (Ourios) carry a project-specific meaning that may differ from the industry-default. Cross-references in italics point to other entries here.

Audit event. A structured record emitted by the miner every time a template is widened (parameters generalised), merged with another template, or versioned. Audit events are themselves stored as logs and are queryable. They are the trail by which an operator can answer “did this template silently change yesterday?” See hazards.md H1.

Bit-identical reconstruction. The property that, for any ingested log line, either Ourios can reproduce the exact original byte sequence from what it stored, or the row carries lossy_flag = true. Never an in-between. Tested as a property test against the corpus. See CLAUDE.md §3.3, hazards.md H7.

Body. The free-form text content of a log record. In OTel terms, the body field of a LogRecord. In Ourios storage, the body is either reconstructible from template + params (most rows) or retained verbatim in a dedicated column (lossy rows, parse failures, or tenants who opted in to always-retain).

Compaction. Background process that merges many small Parquet files into fewer large ones, targeting row-group sizes of 128 MB to 1 GB and file sizes of 256 MB to 2 GB. Driven by the small-file hazard (H4).

Confidence. A scalar in [0, 1] assigned by the miner to each matched row, measuring how well the row matched its assigned template. The three-zone model partitions confidence into clean match (≥ threshold), lossy match (floor ≤ x < threshold), and parse failure (< floor). (Ourios) — extends Drain, which is binary-classifying.

Corpus. A collection of anonymised log lines used as test input. Lives under testdata/corpus/. Public LogPAI corpora form the floor; self-collected corpora per deployment archetype are added over time. Reconstruction, template-count convergence, and merge rate are all measured against the corpus on every miner change.

DataFusion. The Apache project providing the query engine Ourios uses. Ingests logical plans, optimises them, executes against Parquet. Ourios extends DataFusion with two custom logical nodes (render, template_id.resolves_to) but otherwise treats it as a black box. DataFusion specifics never leak into the user-facing DSL (H6).

Drain. The 2017 paper (He, Zhu, Zheng, Lyu — ICWS 2017) that introduces a fixed-depth tree algorithm for online log parsing. The basis of the miner. See docs/rfcs/0001-template-miner.md and docs/talks/0001-template-miner.md.

Drain3. The IBM-maintained fork of Drain that adds persistent state, masking, variable-length wildcards, and dynamic thresholds. Some of its extensions are adopted in Ourios; some are explicitly not. RFC 0001 §4 lists the per-extension verdict.

Drift. The phenomenon where a service’s log format changes between deploys, producing new templates that are aliases of older ones. (Ourios) — drift is detected as a first-class query, not an after-the-fact discovery. See H5 and RFC 0001 §6.7.

DSL. The user-facing query language for Ourios logs (RFC 0002). Compiles to DataFusion logical plans; does not expose SQL. Two candidate predicate sublanguages (OTTL-borrowed vs. distanced) and three top-level surfaces (SQL-clause, LogQL-pipe, Insights-verb) are under design.

Floor. The lower bound of confidence below which the miner declares a parse failure. Default ~0.3. Below the floor, the row is stored body-only and parse_failures_total increments. (Ourios) — not present in the Drain paper.

Fsync. The POSIX call that forces buffered writes to durable storage. The WAL fsyncs before acknowledging an OTLP batch. See H3, CLAUDE.md §3.4.

Hazard. A named failure mode that, if not actively mitigated, silently corrupts data or destroys the project’s value proposition. The eight current hazards are catalogued in docs/hazards.md. New hazards are added via a meta: RFC.

Ingester. The Ourios role that receives OTLP over gRPC/HTTP, mines templates, writes to the WAL, and (eventually) flushes to Parquet via the compactor. One half of the ingester/querier binary split.

Length group. The first-level partition in the Drain parse tree: one branch per distinct token count. Drain assumes lines of different length are probably from different call sites and uses length as a cheap initial filter.

Log group. Drain’s term for a template together with the rows that have matched it. A leaf in the parse tree contains a list of log groups.

Lossy. A row whose lossy_flag is set, indicating that reconstruction from template + params may not be byte-identical. Always paired with the original body being retained on that row. See H7.

LogPAI. The benchmark-corpus project for log parsing (github.com/logpai/logparser). Ourios uses LogPAI corpora (HDFS, BGL, Spark, Apache, OpenSSH, Windows) as the public-corpus floor for benchmarks.

Masking. Pre-tokenisation regex rules that replace volatile sub-strings (IPs, UUIDs, numbers) with placeholders before the miner walks the tree. A Drain3 extension. Whether and where Ourios applies masking is a design choice in RFC 0001 §4.

Merge. When the miner widens an existing template to absorb a new line — e.g. replacing a literal token with a wildcard. Every merge fires an audit event. Strict thresholds make merges rare; audit makes them visible. See H1.

Miner. Short for template miner — the Ourios subsystem that runs Drain online over ingested log lines and emits (template_id, params, confidence, lossy_flag) for each row. Lives in the ourios-miner crate. Designed in RFC 0001.

template_id. The miner’s key for a template: a u64 that is Ourios-derived and tenant-local, not an OTLP field and not portable. Two stores — or one store after an RFC 0023 eviction or a re-mint — can give the same log shape different ids, which is why drift is a first-class query (RFC 0010) and aliases exist (RFC 0007). It shares the Parquet record’s flat namespace with genuine OTLP fields (severity_number, trace_id, body), so the name can read like a wire field; it is not one. The portable identity of a template is its string, which the OpenTelemetry ecosystem is converging on as log.record.template (collector-contrib’s drainprocessor, tracking semantic-conventions #1283 / #2064) — see RFC 0050 for how Ourios accepts and reconciles a template that was derived upstream.

OTLP. OpenTelemetry Protocol, the wire format for telemetry data. The Ourios ingest contract: incoming logs are OTLP over gRPC or HTTP. We do not invent our own format.

OTTL. OpenTelemetry Transformation Language, the OTel Collector’s text-based DSL for filtering and mutating telemetry in processor pipelines. Ourios deliberates between borrowing OTTL’s predicate sublanguage and distancing from it (RFC 0002).

Parquet. The Apache columnar file format Ourios uses for on-disk storage. Per-column compression, predicate pushdown via min/max statistics, bloom filters, page indexes. The on-disk truth of the system; local disk is cache and WAL only. See CLAUDE.md §§2.1, 3.6.

Params. The variable parts of a log line that the miner extracts when matching a template. Bounded per-parameter to 256 B by default; overflow spills to the body column. See H2.

Parse failure. A row whose match confidence falls below the floor. Stored body-only; parse_failures_total counter increments.

Predicate pushdown. A query-engine optimisation where filter predicates are applied as early as possible — at the storage layer rather than after a full scan. Parquet’s min/max page statistics make this nearly free for time-range and equality filters. The mechanism by which predicate queries beat zstdcat | grep (B1).

Property test. A test that asserts an invariant over many randomly-generated inputs (typically via proptest). In Ourios: reconstruction is always a property test; the parser round-trips; the miner’s tree operations preserve invariants. See CLAUDE.md §6.2.

Querier. The Ourios role that accepts queries (over the DSL), plans them through DataFusion, scans Parquet, and returns results. Other half of the ingester/querier split.

Reconstruction. The act of producing the original body of a log line from the stored template + params (and, where retained, the captured whitespace state). Subject to the bit-identical guarantee. See H7.

Row group. Parquet’s unit of compression and predicate-pushdown locality — a horizontal partition of rows within a file. Target size 128 MB to 1 GB. Smaller row groups mean faster row-group skip but worse compression and more metadata overhead.

Semconv registry (Ourios). The weaver registry defining every Ourios telemetry name (ourios.* metrics and attributes), from which the ourios-semconv constants (Rust and TypeScript) are generated. Since 2026-08-25 it lives in the shared jensholdgaard/ourios-semconv repo; this repo consumes it at the ref pinned in semconv/REGISTRY_REF (regenerate via just semconv-generate), and each dashboard-plugin repo carries its own pin. RFCs written before the extraction reference the previous in-repo paths, semconv/registry/ and templates/registry/ — those references are historical, not stale.

Similarity. The Drain match score between an incoming line and a log group’s template: the fraction of token positions where the line matches the template (wildcards count as matches). The single most important knob in the system. See RFC 0001 §3.

SUMMARY.md. mdBook’s table-of-contents file (docs/SUMMARY.md) that defines book navigation. Drafts (no link target) appear as greyed-out entries.

Template. The structural pattern of a class of log lines, with variable parts replaced by wildcards. E.g. ERROR db connection failed for user <*>. The miner extracts templates online from raw logs. (Ourios) — every template is scoped per tenant; the same string in two tenants is two templates.

Template id. The identifier of a template within a tenant. Either a hash of the canonical template string or a per-tenant monotonic integer (open question, RFC 0001 §6.1).

Template tree. The Drain parse tree, scoped per tenant. Its shape is root → length group → token-prefix nodes (depth d) → leaf log groups. (Ourios) — Drain assumes one tree; we keep one per tenant (CLAUDE.md §3.7).

Template version. A monotonic integer that bumps when a template’s representation changes (e.g. token order, new wildcard). The logical identity of the template persists across versions via the alias mechanism. See drift, RFC 0001 §6.7.

Tenant. An isolation boundary: a customer, a project, an environment. Every code path that touches data takes a tenant_id; every Parquet file is partitioned by tenant; every template tree is scoped per tenant. Multi-tenancy is not bolted on (CLAUDE.md §3.7).

Thesis-gate. A benchmark goal whose failure on representative corpora invalidates an architectural pillar — meaning the response is an RFC to revisit CLAUDE.md §2, not a tuning sprint. The five thesis-gates are catalogued in docs/benchmarks.md §7.

Threshold (st). The Drain similarity cutoff above which a line is assigned to an existing log group rather than opening a new one. Ourios default ≥ 0.7; values below 0.7 require an RFC (H1, CLAUDE.md §3.1).

Token-prefix node. Drain’s intermediate tree level: branches on the value of the line’s first N tokens (depth d, paper default 3–4). Below it, at the leaf, is a list of log groups.

Truncation marker. The placeholder that replaces an oversized params slot when the per-parameter byte limit is exceeded. The original value spills to the body column. See H2.

WAL. Write-ahead log. The Ourios ingester writes every acknowledged batch to the WAL, fsyncs, and only then acknowledges to the OTLP client. WAL segments are eventually flushed to Parquet by the compactor. The crash-recovery test SIGKILLs the ingester mid-batch and asserts no acknowledged data is lost. See H3, CLAUDE.md §3.4.

Benchmarks

Referenced from CLAUDE.md §6.2 (“regressions block merges”) and from docs/rfcs/0001-template-miner.md §8. Flat-file, living document, parallel to docs/hazards.md. Updated with measured results as they come in.

This document is an honesty contract with ourselves. The thesis (CLAUDE.md §2) claims that Parquet + Drain-derived template mining + DataFusion beats the naive alternative of byte-level compression over flat text. That claim is falsifiable. This file lists the measurements that would falsify it.

The thresholds were pinned before any number was measured; if we miss them on representative corpora, the thesis is wrong and a pillar changes. As of 2026-06-14 the four gating thesis-gates B1, B2, C1, C2 all pass on the §1 hardware baseline (§9.4/§9.6). A1 fails but no longer gates — RFC 0011 (accepted) reclassified the compression-vs-zstd ratio as a recorded diagnostic (its failure is structural; see §2 / the §7 table).

0. How to read this document

Every goal below carries two labels.

• Scope — thesis-gate, tuning-goal, or diagnostic.
  • A thesis-gate failing on representative corpora means a pillar (CLAUDE.md §2) is wrong. The response is an RFC, not a sprint.
  • A tuning-goal failing means the design is sound but the implementation needs work. The response is a PR.
  • A diagnostic is measured and recorded but gates nothing — it characterises a property or guards against regression. A1 was reclassified here by RFC 0011 (accepted); see §2.
• Bar — must-win, should-win, stretch, or informational.
  • must-win — shipping without it is shipping a broken claim.
  • should-win — expected on representative corpora; explained when missed.
  • stretch — aspirational; missing is not a bug.
  • informational — a diagnostic’s bar: the number is recorded for insight, never blocks.

A goal with scope thesis-gate and bar must-win is load-bearing for the whole project. Four of those below are gating — B1, B2, C1, C2, each marked [THESIS]. A1 keeps the [THESIS] tag (a thesis-relevant measurement) but RFC 0011 (accepted) set its scope to diagnostic: it is recorded, not gating (see its section below and the §7 table).

1. Corpora and methodology

Before any goal is meaningful, the corpora and methodology must be pinned — otherwise we will argue about numbers instead of about architecture.

• Public: LogPAI corpora (HDFS, BGL, Spark, Apache, OpenSSH, Windows) — the same corpora the Drain paper reports on. Lets us reproduce published claims as a sanity floor.
  • LogHub HDFS_v1 is the first of these wired in, as a bench-time-fetched corpus for the query gates: .github/workflows/query-bench.yml downloads HDFS_v1.zip from the official Zenodo record (record 8196385, DOI 10.5281/zenodo.8196385, md5-pinned in the workflow), uses the extracted HDFS.log (~1.47 GiB plain text — above §8’s ≥ 1 GiB canonical minimum) in-job, and discards it with the runner. It is never redistributed: not committed (the testdata/corpus/README.md anonymisation gate — LogHub data is explicitly not sanitised), not attached to a release, not uploaded as an artifact; only aggregate numbers leave the job. LogHub’s license notice, included here as it requires: “The datasets are freely available for research or academic work. For any usage or distribution of the datasets, please refer to the loghub repository URL (https://github.com/logpai/loghub) and cite the loghub paper: Jieming Zhu, Shilin He, Pinjia He, Jinyang Liu, Michael R. Lyu. Loghub: A Large Collection of System Log Datasets for AI-driven Log Analytics. In IEEE International Symposium on Software Reliability Engineering (ISSRE), 2023. The above license notice shall be included in all copies.”
• Self-collected (deferred): at least one anonymised corpus per target deployment archetype. Proposed set:
  1. Structured Java/Spring service (well-templated, low entropy).
  2. Go service under Kubernetes (heterogeneous, mid entropy).
  3. Heterogeneous k8s aggregate across many services (high entropy, mixed formats).
• Hardware baseline: a commodity 8 vCPU / 32 GiB RAM host with gp3-class SSD. All must-win numbers are quoted against this baseline; scaling to larger hardware is a separate question. The realised baseline (the baseline-8vcpu-32gib hardware tag, first used for the §9.4 authoritative run) is a dedicated host with 8 dedicated vCPU, 32 GiB RAM, and a local NVMe-class SSD — at or above the spec on every axis, so numbers quoted against the tag satisfy this baseline. It is identified only by the tag.
• Reference system: zstdcat <file.zst> | grep <pattern>. The “naive alternative” the thesis beats or does not beat. Everything is quoted relative to this, not in absolute terms.

Goals quoted below assume this setup. When a goal is measured on a different setup, the measurement is annotated.

2. Compression goals (Category A)

The core claim that template mining does useful work before byte codecs run.

A1 [THESIS] — End-to-end compression ratio vs. zstd-alone

Demoted to a diagnostic (RFC 0011, accepted). A1 is refuted on every corpus class — including the maximally-templated one — for structural reasons, so it no longer gates any RFC’s validated. It is still measured and recorded (§7 table / §9 series) as the columnar queryability premium and a codec-regression guard. The scope, bar, target, and falsifier below are retained as the diagnostic’s reference line — now informational, not gating.

• Scope: diagnostic (RFC 0011; originally thesis-gate).
• Bar: informational (RFC 0011; originally must-win).
• Metric: bytes(raw_corpus) / bytes(ourios_parquet_directory) compared to bytes(raw_corpus) / bytes(zstd_compressed_corpus).
• Target: Ourios ratio ≥ 3× the zstd-alone ratio, on every corpus in §1. Best-case corpora (well-templated services) should show ≥ 10×.
• Falsifier: if any representative corpus yields ≤ 2× improvement over zstd-alone, the template-mining pillar is not pulling its weight on that class of logs. Open an RFC.
• Why recorded (diagnostic, not a bar): CLAUDE.md §2 pillar #2 describes a logical 50–200× reduction (lines → (template_id, params)) whose payoff is query pruning (B1/B2), not on-disk bytes vs a byte codec. A1 tracks the on-disk ratio as the columnar queryability premium + a codec-regression guard; RFC 0011 (accepted) demoted it from a gate to this diagnostic.

A2 — Bytes per line, amortised

• Scope: tuning-goal.
• Bar: should-win.
• Metric: total Parquet bytes for tenant / line count for tenant.
• Target:
  • Structured service logs: ≤ 30 B/line.
  • Heterogeneous k8s: ≤ 100 B/line.
  • Stretch: ≤ 15 B/line on high-repetition corpora.
• Why: makes A1 legible to operators, who think in bytes-per-line, not ratios.

3. Query performance goals (Category B)

Why not zstdcat | grep? Because the query layer is supposed to exploit structure the tree extracted.

B1 [THESIS] — Predicate-pushdown queries

• Scope: thesis-gate.
• Bar: must-win.
• Query shape: count events WHERE tenant=X AND ts BETWEEN t1 AND t2 AND level='ERROR'.
• Reference: zstdcat files_in_range.zst | grep ERROR | wc -l on the same corpus, same time window.
• Target: Ourios ≥ 10× faster at 1 GiB corpus, widening to ≥ 100× at 100 GiB.
• Falsifier: if Ourios is not materially faster than the zstdcat pipeline on predicate queries, DataFusion + Parquet statistics are not delivering on the “skip row groups via footer reads” pillar (CLAUDE.md §2.1). Open an RFC.
• Instruments: B1 is proven structurally (deterministically) by ourios-querier’s rfc0007_1_* tests. The criterion bench crates/ourios-bench/benches/b1.rs adds the wall-clock ratio: a b1/synthetic group (controlled pruning instrument vs. an in-process zstdcat | grep reference) and a b1/real-corpus group (set OURIOS_B1_CORPUS_DIRS to a comma-separated list of corpus dirs; skipped when unset). The real arm runs OTLP corpora only (corpus/otel-demo-v*, which carry real per-record severity): B1’s predicate filters on severity, and the RFC 0006 §3.3 plain-text loader assigns every line a fixed severity (9 / INFO), so a severity predicate over a plain-text corpus has no selectivity and such dirs are skipped with a note. CI runs land via .github/workflows/query-bench.yml on ci-runner — indicative only. The authoritative numbers are the baseline-8vcpu-32gib run of 2026-06-12 (§9.4): PASS at 34.2× / 25.4× on the two ~1 GB OTel-Demo corpora, with exact row-count agreement against the reference pipeline. Open quality improvement (non-blocking): the measured error bands are ultra-thin (11 / 28 rows), which flatters pruning — a denser error band is the remaining methodological wish.

B2 [THESIS] — Template-exact queries

• Scope: thesis-gate.
• Bar: must-win.
• Query shape: SELECT * WHERE template_id = X AND ts BETWEEN ….
• Target: latency proportional to result cardinality, not to corpus size, above a corpus size of ~10 GiB. Concretely: median latency ≤ 200 ms for a query returning 10 000 rows, regardless of whether the corpus is 10 GiB or 10 TiB.
• Falsifier: if template-exact queries scan proportionally to corpus size, template mining is buying compression but not query locality — the inverted-index collapse thesis (CLAUDE.md §2) is wrong in practice. Open an RFC.
• Instruments: B2 is proven structurally (deterministically) by ourios-querier’s rfc0007_2_* test — for a fixed result the scanned row groups + bytes stay flat as the corpus grows. The criterion bench crates/ourios-bench/benches/b2.rs adds the wall-clock view: a b2/synthetic group (result held constant, corpus scaled 1×/10×/50×) and a b2/real-corpus group over real corpora (set OURIOS_B2_CORPUS_DIRS to a comma-separated list of corpus dirs; skipped when unset, since the corpora aren’t committed). Both loader formats feed it: the OTLP/JSON corpus/otel-demo-v* releases and the bench-time-fetched plain- text LogHub HDFS_v1 (§1). Run with cargo bench -p ourios-bench --bench b2. CI runs land via .github/workflows/query-bench.yml on ci-runner — indicative only. The authoritative numbers are the baseline-8vcpu-32gib run of 2026-06-12 (§9.4): PASS — the windowed template-exact scan stays at 1 row group with a flat ~4.2–5.9 ms latency band across every corpus, including the first reading from a second corpus family (LogHub HDFS_v1, 11.2 M rows: 1/14 row groups, 5.92 ms), while the full-span variant grows with corpus size. The formal target speaks above ~10 GiB, which remains a future scale extension; the flat shape holding at 11.2 M rows across two corpus families is the operative evidence.

B3 — Substring queries (the hard case)

• Scope: tuning-goal.
• Bar: must-match; stretch: beat.
• Query shape: SELECT * WHERE body LIKE '%<substring>%' or equivalent.
• Target: not slower than the reference system. Stretch: faster on well-templated corpora by searching the template text rather than every line.
• Why this is only tuning-goal, not thesis-gate: substring search is the case where the tree does not help directly. We are allowed to match the reference system here; losing against it is a bug but not a pillar failure.

4. Miner correctness goals (Category C)

Correctness is not a performance goal, but it belongs here because these are the properties the benchmark harness actually measures on every run.

C1 [THESIS] — Bit-identical reconstruction rate

• Scope: thesis-gate.
• Bar: must-win.
• Metric: of all non-lossy-flagged rows, fraction whose reconstruct(template, params) equals the ingested bytes exactly.
• Target: 100.000%.
• Falsifier: a single row that reconstructs wrong without a lossy flag is a violation of CLAUDE.md §3.3 and a blocker, not a benchmark regression. Accompanied by: the lossy-flagged fraction should be ≤ 5% on structured corpora, ≤ 20% on heterogeneous ones, as a quality signal (not a gate).
• Why this is a thesis-gate: if we cannot promise reconstruction, the honesty contract (lecture §6) collapses.

C2 [THESIS] — Template count convergence

• Scope: thesis-gate.
• Bar: must-win.
• Metric: template count as a function of lines ingested, on a corpus from a single stable service.
• Grain (amended for #444, 2026-07-10): because the metric is defined per stable service, the gate is evaluated per service.name on a multi-service corpus, not on the whole corpus. A corpus passes iff every service with ≥ 1 M lines converges; a single-service (or plain-text <unknown>) corpus is gated on that one service’s exact-millionth-line ratio, reproducing the pre-amendment verdict for historical converged corpora. The whole-corpus ratio is retained as a diagnostic. See RFC 0006 §3.4.3.
• Target: template count grows sub-linearly and plateaus within 2× of its steady-state value by 1 M lines. Steady-state value is corpus-specific but is on the order of 10²–10⁴ templates for a normal service.
• Falsifier: if template count grows linearly with corpus size, Drain has failed to abstract — we are storing one template per line, which means the tree is providing compression only accidentally. That is the inverse of the thesis. Open an RFC.

C3 — Merge rate

• Scope: tuning-goal.
• Bar: should-win.
• Metric: merges_total / lines_ingested.
• Target: ≤ 1 merge per 10⁵ lines on stable corpora, with every merge carrying an audit event. Spikes above this rate are investigated; they usually indicate a new service version.
• Why only tuning-goal: merge rate depends on corpus stability more than on algorithm quality. The auditing is the invariant (§3.1); the rate is a signal.

C4 — Parameter overflow rate

• Scope: tuning-goal.
• Bar: must-win.
• Metric: fraction of rows where any params slot hit the 256 B limit.
• Target: ≤ 1% on representative corpora, per CLAUDE.md §3.2.
• Falsifier (tuning sense): if >1% on a common archetype, either the limit is too tight for that workload or a masking rule is missing. The response is tuning, not an RFC.

5. Ingest goals (Category D)

The hot path must keep up with real deployments; otherwise none of the above matters.

D1 — OTLP → WAL throughput

Recast per-node (RFC 0034; enacted 2026-07-21). D1’s original metric was lines/second/core — an axis the architecture deliberately serializes twice (sequential per-tenant mining — the CLAUDE.md §3.7-scoped trees assign ids first-seen, which must match WAL-order replay, RFC 0001 §3.5.3; and the single durable WAL stream, §3.4) and one that contradicted D1’s own per-node falsifier (§9.19–§9.21). The must-win below is per-node on the §1 baseline class; the old per-core target and the per-tenant single-stream ceiling are retained as recorded diagnostics — informational, gating nothing (the RFC 0011 A1 pattern). Asserting run: §9.23 (PASS).

• Scope: tuning-goal.

• Bar: must-win.

• Metric: lines/second sustained per node on baseline-8vcpu-32gib, multi-tenant load (soak --tenants N with N = cores) through one shared WAL/commit stream, with WAL fsync batched at 100 ms (the CLAUDE.md §3.4 default).

• Target: the asserting run offers exactly 100 000 lines/s per node and must achieve ≥ 99% of offered (the single acceptance rule — pacing loss up to 1% is within the bar), with p99 ingest-ack latency ≤ 200 ms over that same run. Achieved ≈ offered is also the below-saturation proof: a saturated pipeline cannot keep pace with the paced load, and queue-bound latencies at over-offered load are a different regime that does not count (§9.20’s reading).

• Diagnostics (informational, still recorded — RFC 0034):

  • the original per-core target (≥ 100 000 lines/s/core) is retained as the diagnostic’s reference line; §9.23 records 12,490 lines/s/core at the asserting run.
  • the per-tenant single-stream ceiling — the most one service can push into one tenant (≈ 86k lines/s under the §9.20 probe configuration; §9.20/§9.21) — guards the mining path against regression.

  Neither gates any RFC’s validated.

• Falsifier (tuning sense): below this we cannot ingest a meaningful share of production traffic per node, which makes the operational story uninteresting.

D2 — WAL → Parquet compaction keeps up

• Scope: tuning-goal.
• Bar: must-win.
• Metric: WAL backlog (bytes, segments) as a function of time under sustained ingest at D1’s rate.
• Target: bounded; backlog returns to zero during any one-hour window of sustained load.
• Falsifier (tuning sense): a growing backlog under steady-state load means compaction is the bottleneck — a correctness-adjacent bug because it lets the WAL grow unboundedly.

D3 — Small-file count under sustained load

• Scope: tuning-goal.
• Bar: should-win.
• Metric: number of Parquet files per tenant per day after background compaction has settled.
• Target: file sizes cluster in the 256 MiB–2 GiB band per CLAUDE.md §4 / hazard 4. Fewer than 5% of files below 128 MiB at steady state.
• Why: the small-file problem is a named hazard, not a nice-to-have.

6. Honesty goals (Category E)

Not performance. Not falsifiable by a benchmark in the usual sense. Listed here because the benchmark harness asserts them on every run.

E1 — Zero silent merges

• Scope: correctness invariant (not a benchmark).
• Metric: in the corpus-test suite, for every row whose template_id changed over its lifetime in the tree, an audit event exists with matching timestamp and tenant.
• Target: 100%. This is a proptest, not a measurement.

E2 — Zero cross-tenant leakage

• Scope: correctness invariant (not a benchmark).
• Metric: no template mined under tenant A ever appears in tenant B’s tree or in a row for tenant B.
• Target: 100%. Asserted via corpus tests that interleave lines from two synthetic tenants and verify complete isolation.

7. The thesis-gate summary

The five [THESIS]-tagged goals, consolidated:

Policy: if one thesis-gate fails on one representative corpus, that is a corpus-specific tuning RFC. If two or more thesis-gates fail on any representative corpus, that is a pillar-level RFC — we pause implementation and revisit CLAUDE.md §2 before continuing.

This escalation rule is the point of the whole document. The worst failure mode for a greenfield project is shipping something whose central claim quietly fails on real data and then papering over it with more implementation. These goals exist so we cannot do that to ourselves without noticing.

8. What is deliberately out of scope

• SIEM-style full-text search latency — explicitly out of scope (CLAUDE.md §1).
• Cross-tenant aggregation queries — tenancy is isolation-first (CLAUDE.md §3.7). Aggregations that cross tenants are an RFC topic, not a benchmark.
• LLM-based parser comparisons — interesting, deferred. Listed in RFC 0001 §7 as an alternative. Benchmarking it would be a separate RFC.
• Cold-start query latency — below a corpus size of ~1 GiB the overhead of Parquet metadata dominates, and the thesis is uninteresting. Benchmarks start at 1 GiB.

9. Status

First measurements landed 2026-06-01 (the writer-side gates A1 / C1 / C2 — see §9.1). They are diagnostic, not canonical: they ran on a GitHub-hosted runner (ci-runner), not the §1 hardware baseline (baseline-8vcpu-32gib), against an OTel-Demo corpus that is shape-representative (real multi-service template + envelope diversity) but not size-representative — every corpus is well below §8’s ≥ 1 GiB canonical minimum, so this run is intentionally diagnostic, not a thesis verdict. The query-side gates now have instruments — B1 and B2 are proven structurally in ourios-querier, and both have criterion latency benches with real-corpus arms (§B1/§B2 “Instruments”; OTel-Demo for B1, OTel-Demo + the bench-time-fetched LogHub HDFS_v1 for B2, run on ci-runner via .github/workflows/query-bench.yml as indicative numbers). 2026-06-11 extended the writer-side scale series to ~1 GB (§9.2) and landed the first B1/B2 query readings (§9.3) — recorded here as indicative ci-runner entries per the maintainer’s 2026-06-12 authorization. 2026-06-12 landed the authoritative baseline run (§9.4): every gate measured on the §1 hardware (baseline-8vcpu-32gib), recorded per the maintainer’s 2026-06-12 authorization. B1, B2, C1, and C2 pass authoritatively; on that basis RFC 0007 flipped to validated (its gates, per docs/verification.md §3, are the querier-pillar ones — B1/B2). A1 fails authoritatively and carries a hardware-sensitivity caveat (§9.4). (A1 was subsequently reclassified a recorded diagnostic, not a gate — RFC 0011, accepted 2026-06-14. The A1 readings throughout §9 are diagnostic; A1 gates nothing, and the “open gate” / “must-win” framing in the dated entries below is superseded.)

Reviewers: a PR that materially affects the hot path must either (a) cite the benchmark result and its delta against the relevant goal, or (b) explain why the hot-path effect is bounded below measurability. “I did not run the benchmarks” is a PR rejection, per CLAUDE.md §6.6.

No ourios-bench --update-benchmarks-md run has populated this region yet. It is the bench-managed results area — automated runs replace everything between these markers with one table per (git-sha, hardware). The hand-written §9.1 below is the curated diagnostic narrative and lives outside the region so automated runs never touch it. (This empty region is pre-placed so the first --update-benchmarks-md run replaces it in place rather than appending a second results section at end-of-file.)

9.1 Results — 2026-06-01 (diagnostic, ci-runner)

Corpus. corpus/otel-demo-v{1..4} — OTel Demo 2.2.0 logs captured via the collector fileexporter (workflow .github/workflows/capture-otel-demo-corpus.yml), business-service logs only (collector self-telemetry + load-generator filtered out), OTLP/JSON. Sizes 30 / 136 / 272 / 547 MiB — all below §8’s ≥ 1 GiB canonical benchmark minimum (this run is deliberately sub-minimum, to chart the trend, hence diagnostic). Hardware. ci-runner (hosted, ~4 vCPU) — not the §1 baseline, so deltas are indicative, not authoritative.

A1 — compression (target: ourios ≥ 3.0× zstd-19).

Scale series (ourios at the production ZSTD-3 default):

Codec sweep (v4 = 547 MiB, ourios ZSTD level varied):

A1 verdict: FAIL (target 3.0×; best observed 0.829). Both levers are bounded. Scale lifts the delta but plateaus ~0.78 (ourios asymptotes ~25×; zstd-19 is flat ~32× — the logs are locally repetitive, so zstd compresses them well at any size, not via a whole-corpus window). Raising ourios’s codec to ZSTD-19 adds only ~+0.07 and saturates by level 9. Even at equal codec strength, ourios stays ~17% larger than monolithic zstd-19: a structural cost of columnar Parquet (per-column/per-chunk framing, page indexes, row-group metadata, bloom filters) versus zstd-19 over one concatenated stream. That same chunking is what enables row-group skipping — so the ~17% space premium is the price of queryability, not an optimisation target. On pure compression of this corpus, ourios ≈ 0.83× zstd-19; the thesis rests on query performance (B1/B2), not on beating a byte codec.

C1 — reconstruction (target: 100% bit-identical or flagged lossy). PASS at every size: 1.0 reconstruct rate, ~1.1% of records flagged lossy (structured/kvlist bodies) and retained verbatim per CLAUDE.md §3.3.

C2 — template-count convergence (target: sub-linear). PASS (supportive). Templates grew 282 → 429 → 722 → 1322 while records grew 38k → 183k → 366k → 735k — sub-linear throughout. The formal gate abstains below 1 M lines (§3.4.3), but the curve shape is the strongest evidence yet for the template-mining premise.

Escalation (§7). One gate (A1) fails, on a size-non-representative corpus (all < §8’s 1 GiB minimum) and non-baseline hardware — so this is “corpus-specific,” not the two-gate pillar-level pause. C1 + C2 support the thesis. The production ZSTD-3 default is retained: the codec gain is small, saturates by level 9, and the residual gap is structural, so a higher default isn’t worth the ingest-CPU.

9.2 Results — 2026-06-11 (diagnostic, ci-runner) — A1 / C1 / C2 at ~1 GB

Corpus. corpus/otel-demo-v5 (1,042,274,219 B) and corpus/otel-demo-v6 (1,034,615,505 B) — same capture pipeline as §9.1, extending the scale series to ~1 GB (both within 4% of, but still just under, §8’s ≥ 1 GiB binary minimum). v6 was captured with the OTel Demo failure flags enabled (adFailure cartFailure productCatalogFailure), so it carries a real error band; v5 is an unflagged capture. Hardware. ci-runner — indicative, not the §1 baseline. Runs. bench.yml 27370641352 (v5), 27373716667 (v6).

A1 — compression (target: ourios ≥ 3.0× zstd-19).

A1 verdict: FAIL (target 3.0×). The scale series now reads 0.465 (v1, 30 MiB) → 0.666 (v2) → 0.725 (v3) → 0.758 (v4) → 0.828 (v5) / 0.824 (v6): the delta is size-driven and still rising, but decelerating — the crossover is not reached at ~1 GB, consistent with §9.1’s structural reading (ourios asymptotes ~26×; zstd-19 stays flat ~32×). v5 ≈ v6 shows the failure-flag error band does not perturb A1. This is the first A1 miss at (essentially) canonical size, so §9.1’s “size-non-representative” mitigation no longer applies; it remains a single-gate fail (no §7 two-gate pause), the §9.1 structural explanation stands, and the thesis-deciding counterpart — B1/B2 — now passes indicatively (§9.3). Whether the §7 corpus-specific tuning-RFC response triggers is a maintainer decision, sensibly taken once an authoritative baseline-8vcpu-32gib run confirms the number. (Resolved 2026-06-12: the §9.4 baseline run confirms — and slightly worsens — the deltas; the decision is now live with the maintainer.)

C1 — reconstruction (target: 100% bit-identical or flagged lossy). PASS on both: 1.000000 — v5 reconstructs 1,213,004 / 1,213,004 non-lossy rows exactly (lossy ratio 0.0114); v6 1,208,323 / 1,208,323 (lossy 0.0112).

C2 — template-count convergence (target: ratio ≥ 0.5 at 1 M lines). PASS on both — and for the first time on ≥ 1 M-line corpora, so the formal gate applies rather than §9.1’s abstention: v5 convergence ratio 0.756 (end count 1605, sample cadence 1336); v6 ratio 0.760 (end count 1606, cadence 1329).

9.3 Results — 2026-06-11 (indicative, ci-runner) — first B1 / B2 query readings

Corpus. corpus/otel-demo-v{4,5,6} (the §9.1 / §9.2 captures). The LogHub HDFS_v1 B2 arm did not run (fetch_hdfs off — memory-bound on the hosted runner), so only one corpus family has fed the query gates. Hardware. ci-runner — indicative, not the §1 baseline. Runs. query-bench.yml 27379085890 (B1 + the B2 structural metrics, after the effective-timestamp stack #178/#179) and 27357104694 (the prior run; its windowed / full-span latencies are quoted where noted). Recording. B1/B2 entries land in §9 per the maintainer’s 2026-06-12 authorization. RFC 0006 never reserved §9 (its §1 anticipated B1/B2 landing “in a follow-up extension PR once the querier is live” — RFC 0007); the workflow itself never writes §9 — every entry here is curated by hand.

B1 — predicate pushdown vs zstdcat | grep (target: ≥ 10× at 1 GiB). Query: severity ERROR, full corpus span. Run 27379085890:

Row counts agree exactly with the reference pipeline on both corpora. v4 is skipped: the unflagged 100-user capture genuinely contains zero error-band rows, so the predicate selects nothing.

B1 verdict: PASS (indicative) — both corpora clear the ≥ 10× bar at 3–4× margin, on the first real-corpus reading. Caveats, stated plainly: ci-runner, not the §1 baseline; the error bands are ultra-thin (11 / 28 rows — extreme selectivity is the friendliest case for pruning); both corpora sit just under the §8 1 GiB minimum. An authoritative baseline-8vcpu-32gib rerun (ideally with a denser error band) is required before this counts as the canonical B1 number.

B2 — template-exact latency ∝ result, not corpus. Windowed 1-hour template-exact query, result roughly constant as the corpus grows. Structural metrics (run 27379085890): scanned row groups stay flat at 1 — v4 1/5, v5 1/6 (17,632 rows, 1.86 MB), v6 1/6 (11,750 rows, 1.59 MB). Wall-clock (prior run 27357104694): windowed latencies sit in a flat ~3.4–4.1 ms band (v4 3.59 / v5 4.13 / v6 3.40 ms) while the full-span variant grows with corpus size (7.3 / 10.6 / 10.6 ms) — exactly the result-bound-vs-corpus-bound split the gate asks for.

B2 verdict: PASS (supportive, indicative) — the flat shape is confirmed on real corpora at ~1 GB; the formal target speaks above ~10 GiB, which remains unmeasured, and the second corpus family (HDFS_v1) hasn’t fed the arm yet.

RFC 0007 validated assessment. These are the measurements the RFC 0007 green → validated gate needs, but not yet in the form the ladder requires (§1 quotes must-win numbers against baseline-8vcpu-32gib): see the status note in docs/rfcs/0007-querier.md. The RFC stays green with a validated-pending note — authoritative baseline rerun required; denser error band and a second corpus family supporting. (Resolved 2026-06-12: the §9.4 authoritative run delivered the baseline rerun and the second corpus family (HDFS_v1); RFC 0007 is validated. The denser error band remains an open quality improvement.)

9.4 Results — 2026-06-12 (authoritative, baseline-8vcpu-32gib)

Corpus. corpus/otel-demo-v{1..6} (the §9.1 / §9.2 captures; 30 MiB → ~1 GB) for A1 / C1 / C2 and B1/B2’s OTel-Demo arms, plus — for the first time — the bench-time-fetched LogHub HDFS_v1 (§1; ~1.47 GiB plain text, 11,175,629 rows ingested across 5 files) feeding the B2 arm as the second corpus family. Hardware. baseline-8vcpu-32gib — the §1 baseline (8 dedicated vCPU, 32 GiB RAM, local NVMe-class SSD). These are the authoritative numbers the §1 methodology quotes must-win gates against; the §9.1–§9.3 ci-runner entries remain indicative history. Runs. Dedicated baseline host (no CI run id): one ourios-bench run per corpus (A1/C1/C2) plus one query-bench run (B1 + B2), executed 2026-06-11/12; raw logs retained by the maintainer. Recorded per the maintainer’s 2026-06-12 authorization.

A1 — compression (target: ourios ≥ 3.0× zstd-19).

A1 verdict: FAIL (authoritative) (target 3.0×; best observed 0.751). The delta is monotonic with corpus size and the crossover is unobserved, consistent with §9.1’s structural reading. One finding must be recorded honestly: the authoritative deltas sit below the ci-runner series (0.465 → 0.828) at every size — the ourios side compressed less effectively on this hardware (e.g. v5: 23.8× vs CI’s 26.3×) while zstd-19 stayed essentially stable (31.7× on both) — i.e. the ourios writer’s output is environment-sensitive (suspected row-group sizing / threading effects on the resulting encodings). That is now an open A1 investigation item alongside the structural gap itself. A1 gates the compression pillar (RFC 0006’s remit); the §7 escalation response is with the maintainer.

C1 — reconstruction (target: 100% bit-identical or flagged lossy). PASS (authoritative) on every corpus: 1.000000 throughout — v5 reconstructs 1,213,004 / 1,213,004 non-lossy rows exactly (lossy ratio 0.0114), v6 1,208,323 / 1,208,323 (lossy 0.0112); v1–v4 likewise 1.000000 (lossy 0.0097–0.0112). The formal ≥ 1 M-line gate passes on the baseline.

C2 — template-count convergence (target: ratio ≥ 0.5 at 1 M lines). PASS (authoritative) on both ≥ 1 M-line corpora: v5 ratio 0.756 (end template count 1605, sample cadence 1336), v6 ratio 0.760 (end count 1606, cadence 1329). v1–v4 abstain (< 1 M lines), as in §9.1.

B1 — predicate pushdown vs zstdcat | grep (target: ≥ 10× at 1 GiB). Query: severity ERROR, full corpus span. v4 is skipped (zero error-band rows, as in §9.3).

Row counts agree exactly with the reference pipeline on both corpora (11 and 28).

B1 verdict: PASS (authoritative) — both corpora clear the ≥ 10× bar at 2.5–3.4× margin on the §1 baseline. Remaining caveat, non-blocking: the error bands are still ultra-thin (11 / 28 rows — the friendliest case for pruning); a denser error band stays an open quality improvement.

B2 — template-exact latency ∝ result, not corpus.

Full-span template-exact (result grows with the corpus, so latency may too):

Windowed 1-hour template-exact (the gate’s shape: result roughly constant as the corpus grows):

The HDFS_v1 row is the first reading from the second corpus family (plain-text, the template-diversity case): the corpus is 8–15× the OTel-Demo row counts, yet the windowed scan still touches 1 row group (13 pruned) and stays inside the same flat latency band, while the full-span variant grows with the corpus (6.84 → 30.19 ms) — exactly the result-bound-vs-corpus-bound split the gate asks for.

B2 verdict: PASS (authoritative) — windowed ~10–28 k-row results answer in 4.2–5.9 ms (gate: ≤ 200 ms for ~10 k rows), flat from 735 k to 11.2 M rows across two corpus families. The formal target’s ≥ 10 GiB regime remains a future scale extension; the measured shape is the operative evidence.

RFC 0007 green → validated (resolved). The docs/verification.md §3 ladder reads: “Every thesis-gate in benchmarks.md §7 that the RFC’s pillars touch passes on representative corpora.” RFC 0007’s pillar is the query engine (pillar #3); its gates are B1 and B2, both now passing authoritatively on the §1 baseline over ~1 GB+ corpora including a second family. A1 does not gate RFC 0007 — it belongs to the template-mining/compression pillar, measured under RFC 0006. RFC 0007 is therefore flipped to validated (see its status note); accepted awaits maintainer sign-off per the ladder.

9.5 Results — 2026-06-13 (diagnostic, local unknown hardware) — A1 / C1 / C2 on HDFS_v1

Corpus. LogHub HDFS_v1 (Zenodo record 8196385, md5 76a24b4d…) — 11,175,629 lines, 1,577,982,906 raw bytes; fetched at bench time, never redistributed (query-bench.yml). The maximally-templated log corpus (a handful of templates over 11.2 M lines) — the single best case for the template-mining compression premise. Run via ourios-bench --gates a1,c1,c2 --parquet-zstd-level 19 --allow-unknown-hardware. Local hardware → diagnostic, not authoritative; A1’s verdict is corpus-structural and hardware-independent (compressed bytes are deterministic), C1/C2 are ratios, so the findings hold regardless of the runner.

A1 — the decisive finding (→ RFC 0011). A1 had only ever been measured on OTel-Demo (best 0.829×, §9.1/§9.4). HDFS_v1 is the corpus that should most reward template mining, yet A1 fails harder (0.516×): the more templated the corpus, the more completely monolithic zstd-19 captures its redundancy in one window (16×), while template mining’s extracted params (block IDs, timestamps, IPs) are high-cardinality columns that don’t compress as well and the columnar layout adds framing. The best case for template mining is the best case for the byte codec. So ≥ 3× over zstd cannot hold on any realistic log corpus — A1 is demoted to diagnostic and template mining’s compression value is recognised as logical/query-pruning (B1/B2), not on-disk bytes. See RFC 0011.

C1 + C2 — the miner pillar’s real gates, PASS on a representative corpus. At 11.2 M lines C1 is bit-identical (1.0) with a 4.6e-06 lossy ratio, and C2 plateaus at 40 templates with the formal gate applying (not abstaining, unlike the §9.1 sub-1 M runs). Under RFC 0011 these are RFC 0001’s validated thesis gates — both pass here. The authoritative baseline-8vcpu-32gib representative rerun (for the actual RFC 0001 validated flip) followed on 2026-06-14 (§9.6); as expected of deterministic verdicts, the numbers are identical.

9.6 Results — 2026-06-14 (authoritative, baseline-8vcpu-32gib) — C1 / C2 on HDFS_v1

Corpus. LogHub HDFS_v1 (Zenodo record 8196385, md5 76a24b4d…) — 11,175,629 lines, 1,577,982,906 raw bytes; fetched at bench time on the baseline host, md5-verified, never redistributed. Hardware. baseline-8vcpu-32gib — the §1 baseline (8 dedicated vCPU, 32 GiB RAM, local SSD), provisioned for this run and torn down immediately after. These are the authoritative C1 / C2 numbers for RFC 0001’s validated gates. Run. Dedicated baseline host (no CI run id): one ourios-bench --gates c1,c2 --hardware-kind baseline-8vcpu-32gib run at git 9a57ace; results JSON retained by the maintainer (2026-06-14T00-36-23.225Z-9a57ace.json). A1 was deliberately not run — it is diagnostic, not gating (RFC 0011); the §9.5 diagnostic A1 reading stands.

Authoritative confirmation. The verdicts match §9.5’s local diagnostic run bit-for-bit — expected, since C1 (reconstruction fidelity) and C2 (template-count convergence) are deterministic functions of (corpus, miner) with no wall-clock or hardware-sensitive component (contrast A1’s writer-environment sensitivity, §9.4). The value of this run is the authoritative hardware_kind stamp on the two gates that, under RFC 0011, define RFC 0001’s validated: both PASS on a representative ≥ 1 M-line corpus on §1 baseline hardware.

9.7 Results — 2026-06-15 (authoritative, baseline-8vcpu-32gib) — D2 / D3 / B2-post (RFC 0009 compaction)

Hardware. baseline-8vcpu-32gib — the §1 baseline (8 dedicated vCPU, 32 GiB RAM, local SSD), provisioned for this run and torn down immediately after. These are the authoritative D2 / D3 / B2-post numbers for RFC 0009’s validated measure (RFC0009.7). Run. Dedicated baseline host (no CI run id): the ourios-bench compaction bench at git 4d52288. Two invocations — the band-scale one-shot (OURIOS_COMPACTION_BASELINE=1, FILES=32, ROWS=4800, BODY_BYTES=4096) for D2/D3, then the b2-post-compaction criterion group. Synthetic (no corpus): D2/D3 drive one partition of 32 small files (~485 MiB) through compact_partition; B2-post queries 32-files-vs-1-file with the result set held constant.

Reading. D3 is the headline: a band-scale compaction lands its output squarely in the H4 256 MiB–2 GiB target with zero sub-128 MiB files — the small-file problem, eliminated. D2 shows consolidation runs at ~167 MiB/s on one partition/thread, far above any plausible per-partition seal rate, so a backlog drains (the “keeps up” property). B2-post quantifies the query payoff that motivated RFC 0009 (the PR #92 B2 finding that per-file footer/metadata reads dominate): collapsing 32 files → 1 cuts the footer reads ~6× on this query. The structural reductions (32 → 1 files / row groups, rows conserved) are hardware-independent and also pinned in ourios-parquet’s rfc0009_1_* / compaction_conserves_every_row tests; these wall-clock figures are the baseline-hardware stamp for RFC 0009’s validated. The full sustained-ingest soak (D2’s “backlog returns to zero in a one-hour window at D1’s rate”) and D1 itself remained unrun until §9.19 (2026-07-20: D2 soak PASS; D1’s per-core bar is the open reading) — the throughput here is the RFC0009.7 D2 measure, not that soak.

9.8 Results — 2026-06-18 (authoritative, baseline-8vcpu-32gib) — ingest write-path + recovery (criterion) and real-corpus A1 / C1 / C2 + B1 / B2

Hardware. baseline-8vcpu-32gib — the §1 baseline (8 dedicated vCPU, 32 GiB RAM, local SSD), provisioned for this run and torn down immediately after. Two such hosts (one per invocation set), both at git d3f2cae. Run. (a) the self-contained ourios-bench criterion benches ingest_write_path (RFC 0014) and recovery (RFC0008.3) — synthetic, no corpus — at full criterion settings; (b) the ourios-bench binary --gates a1,c1,c2 against two real corpora, plus the b1/b2 criterion benches (--warm-up-time 1 --measurement-time 3, matching query-bench.yml) over those corpora. Corpora: LogHub HDFS_v1 (Zenodo record 8196385, md5 76a24b4d… — 11,175,629 lines / 1,577,982,906 raw bytes (1.47 GiB) of real Hadoop production logs, above §8’s ≥ 1 GiB canonical minimum) and the frozen OTel-Demo v1 (corpus/otel-demo-v1, 38,782 lines / 31.5 MiB). HDFS is fetched in-job and never redistributed (§1).

(a) Ingest write path + recovery — supportive wall-clock (criterion).

Single-threaded micro-benches on synthetic records — supportive wall-clock (the structural sides are pinned by ourios-ingester’s RFC 0014 / ourios-wal’s RFC0008.3 tests), not gates. Dedicated hardware ran ~20–30% faster with much lower variance than the indicative ci-runner figures.

(b) Thesis gates A1 / C1 / C2 on real corpora.

C1 reconstructs every non-lossy row bit-for-bit across 11 M real production lines — the §3.3 invariant holds on real data at scale. C2 converges on HDFS (40 templates over 11 M lines; ratio 0.825 ≥ the threshold) — the template-mining thesis on a real corpus. A1 fails as expected: it is a recorded diagnostic, not a gate (RFC 0011) — template mining’s value is query pruning (B1/B2), not on-disk bytes beating a whole-stream codec.

(c) Query gates B1 / B2 on real corpora.

The windowed HDFS arm is the headline: a time-bounded query on the real 11 M-line corpus prunes 13 of 14 row groups via Parquet min/max statistics — the predicate-pushdown thesis (pillar #1) on real production data, ~5× faster than the unwindowed scan. (B1’s real-corpus arm skipped: OTel-Demo v1 has no error-band severity_text rows for the selectivity probe.) B1/B2’s structural pruning is the gate (pinned in ourios-querier); these are the baseline-hardware wall-clock stamp.

Not committed by the bench tooling — this is the curated narrative; the managed BENCH-RESULTS region above is for --update-benchmarks-md runs. The b1/b2 criterion timings use the reduced --warm-up-time 1 --measurement-time 3 (matching query-bench.yml); the structural pruning/template numbers are exact and criterion-setting-independent.

9.9 Results — 2026-07-03 (indicative, ci-runner) — B1 / B2 post-RFC 0022 (promoted attribute columns)

Purpose. The RFC 0022 §5 RFC0022.5 note: the promoted-attribute write path (per-key resource.<k> / attr.<k> columns + the two-arm predicate compile) must leave B1/B2 unchanged. This is the indicative re-run after RFC 0022 went green (#345–#348); the pruning counters are pinned structurally in crates/ourios-querier/tests/rfc0022_attr_columns.rs, this entry is the wall-clock stamp. Corpus. corpus/otel-demo-v4 (107,332 records → 735,377 mined rows / 5 files) and corpus/otel-demo-v5 (163,929 records, ~1.04 GB raw → 1,367,532 mined rows / 6 files). The LogHub HDFS_v1 arm did not run (fetch_hdfs off — memory-bound on the hosted runner). Hardware. ci-runner — indicative, not the §1 baseline. Run. query-bench.yml 28686650566 at git 6e3301b (the RFC 0022 green merge).

B1 — predicate pushdown vs zstdcat | grep (target: ≥ 10× at 1 GiB). Query: severity ERROR, full corpus span.

Row count agrees exactly with the reference pipeline. v4 is skipped as in §9.3 (its capture has no error-band rows).

B1 verdict: PASS (indicative), no regression — 34.8× against §9.3’s 40.0× on the same corpus, comfortably inside hosted-runner noise and 3.5× above the bar. Same caveats as §9.3: ultra-thin error band, corpus just under the §8 minimum, not the §1 baseline.

B2 — template-exact latency ∝ result, not corpus.

B2 verdict: PASS (supportive, indicative), no regression — the windowed latencies sit in the same flat few-ms band as §9.3/§9.8 while the full-span variants grow with corpus size, and everything is orders of magnitude under the 200 ms bar. The formal target speaks above ~10 GiB, which remains unmeasured on this runner class.

Assessment. The promoted-column machinery (extra column chunks per row group on the write side; the two-arm OR compile on the read side) shows no measurable drag on either gate. The RFC 0022 green → validated step still requires the authoritative baseline-8vcpu-32gib rerun per the standing bench policy (maintainer opt-in); this entry is its indicative precursor, curated by hand as in §9.3 — the workflow never writes §9.

9.10 Results — 2026-07-04 (authoritative attempt, baseline-8vcpu-32gib) — B1/B2 at 16 GiB: run blocked, miner finding

Purpose. The first run in the §8 10–100 GiB band: B2’s formal target speaks above ~10 GiB and had never been measured there. Corpus. LogHub HDFS_v2 (bench-time fetch, never redistributed): 31 files, 17,240,888,465 bytes ≈ 16.1 GiB raw, ~71 M lines of Hadoop daemon logs — the first corpus in our set whose shape (stack traces, multi-format node logs) differs qualitatively from HDFS_v1’s block events. Hardware. baseline-8vcpu-32gib, provisioned for the run and torn down after. Outcome: the run did not complete — it produced a product finding instead. The B2 store build was OOM-killed at 31.5 GiB RSS: the miner mints templates without bound on this corpus shape (template ids ≥ 56,199 by the 1.8 GiB subset mark, busiest template covering 0.67 % of 8.37 M rows; memory ~linear at ≈2× corpus bytes). Two bench-side pathologies were found and fixed en route — the eager corpus load (#350, now streaming: 1.3 GiB flat over hours) and a quadratic harness snapshot capture (#351, ~400× store-build speedup; gdb stacks exonerate the miner’s CPU path). RFC 0023 (bounded template memory) is the response; its RFC0023.7 criterion is this exact run completing.

What did land before the kill (recorded as diagnostic):

B2’s shape — flat windowed latency, window-driven pruning — holds wherever memory allows; the fragmentation itself (56 k templates, busiest at 0.67 %) also means pillar #2’s logical reduction fails on this corpus shape, which is the same finding from the pruning side. B1 did not reach its arms (stopped before the reference build once the OOM trajectory was clear). No gate verdict is claimed from this entry; the §8-band verdict waits on RFC 0023 + the rerun.

9.11 Results — 2026-07-04 (authoritative, baseline-8vcpu-32gib) — B1 / B2 at 16 GiB + RFC0023.7

Purpose. The §8 10–100 GiB band’s first completed measurement (the §9.10 attempt OOM’d), doubling as RFC0023.7 (bounded mining must complete this exact corpus under 8 GiB peak RSS) and the first B1/B2 readings at ≥ 10 GiB — where B2’s formal target speaks. Corpus. LogHub HDFS_v2 (bench-time fetch): 31 files, 17,240,888,465 bytes ≈ 16.1 GiB, 71,116,785 mined rows → 21 files / 80 row groups. B2 ran under the §3.3 Fixed severity baseline; B1 under the opt-in OURIOS_CORPUS_SEVERITY=log4j extraction (#350), stated per the methodology rule. Hardware. baseline-8vcpu-32gib, provisioned for the run, torn down after. Git 19e0886 (RFC 0023 bounds + telemetry merged).

RFC0023.7 — bounded mining at scale: PASS. Peak RSS 1.73 GiB across both benches’ store builds (5 s sampler), vs the §9.10 OOM at 31.5 GiB on identical input — an 18× reduction, under the 8 GiB bar with 4.6× headroom. Both benches completed (B2 phase 35 min; B1 including its zstd-19 reference build ~2.8 h).

B1 — predicate pushdown vs zstdcat | grep (target: ≥ 10× at 1 GiB, widening to ≥ 100× at 100 GiB). Query: severity ERROR, full 16 GiB span.

Row count agrees exactly with the reference pipeline.

B1 verdict: PASS (authoritative) — the ≥ 100× mark projected for 100 GiB is crossed at 16 GiB. With §9.8’s ~35–40× at ~1 GiB, the measured trajectory confirms the widening the target predicted: the reference’s cost grows with corpus bytes while Ourios’s grows with the matching row groups.

B2 — template-exact latency ∝ result, not corpus (formal target: ≥ 10 GiB, ≤ 200 ms for 10 k rows).

B2 verdict: PASS (authoritative, first ≥ 10 GiB reading) — the windowed query answers in the same few-ms band as the ~1 GiB corpora (§9.3/§9.8/§9.9): latency tracks the result, not the 71 M-row corpus.

The fragmentation datum (§9.10’s open question, quantified). The “busiest template” is id 0 — NO_TEMPLATE: under the default 20 k ceiling, ~79 % of HDFS_v2’s rows took the §6.3 parse-failure path (bodies retained bit-faithfully; observable via ourios.miner.parse_failure.reason, RFC0023.6). Template mining contributes little on this corpus shape — and the B1/B2 numbers above show the floor it degrades to (first-class-column + time pruning over Parquet statistics) still clears every gate. Follow-up noted: the B2 bench’s busiest-template picker should exclude NO_TEMPLATE so the full-span arm measures a true template-exact query on such corpora.

Assessment. RFC 0023’s §5 is fully discharged (this entry is the .7 record); the RFC flips red → green alongside this entry. The §8-band thesis verdict on real, hostile-shaped production logs: pruning compounds with scale (B1), result-bound latency holds (B2), and the mining-fragmentation failure mode is now bounded, observable, and priced.

9.12 Results — 2026-07-09 (indicative, local M-series) — otel-demo v8 capture: C1 / C2

The run is dated 2026-07-09; its C2 verdict was re-scored under the per-service gate on 2026-07-10 (#444 / RFC 0006 §3.4.3), so the resolution dates below post-date the heading.

Corpus. corpus/otel-demo-v8 (published GitHub release): a 48-hour OTel-Demo 2.2.0 capture at 150 locust users with the adFailure + paymentFailure feature flags active — 690,355 OTLP LogsData batches / 4,948,596 log records / 2.96 GB uncompressed, the largest and most hostile real capture to date (deliberately injected failure modes, multi-service, long-horizon). Calibration manifest at testdata/calibration/otel-demo-v8.json (RFC 0024 §3.1).

C1 — bit-identical reconstruction: PASS, perfect. The corpus holds 4,948,596 records (the calibration manifest’s count); 17 of them (all kafka, 0.0003 %) took the §3.3 lossy-flag path with their bodies retained, and C1 = 1.000000 over the remaining 4,948,579 rows — the honesty contract holds at 4.9 M rows through failure-mode churn.

C2 — template-count convergence (bar: ratio ≥ 0.5 at 1 M lines, evaluated per service since #444): PASS. Under the per-service gate (RFC 0006 §3.4.3, amended 2026-07-10) the corpus passes: the only service that clears the 1 M-line evaluation floor is cart, which converges at ratio 1.000 with two templates. Every other service abstains for want of volume; the whole-corpus ratio (0.199, end template count 14,631, sample cadence 4,833) is retained below as a diagnostic — it is a category error to grade a multi-service corpus as one Drain stream (§3.4.3 rationale). The per-service decomposition (splitting on service.name and re-running the gates per service) localises the whole-corpus fragmentation completely:

The gate folds over the gated services (those ≥ 1 M lines): cart is the sole such service and it passes, so the corpus passes. cart clears the formal gate at 2.76 M lines with two templates; the smaller services abstain below the 1 M-line floor, so they are not graded — though their observed counts (1–17 templates over 0.5–1.0 M lines) sit at the same near-flat convergence. The kafka broker, also abstaining, is the outlier: it mints 14,608 templates on 2.8 % of the lines. Mechanism (measured): kafka’s cleaner logs emit 3-token lines whose third token is a unique offset-bearing path (Deleted log /tmp/kafka-logs/…/00000000000000000429.log.deleted., 11,651 distinct) — one varying token in a 3-token line is similarity 2/3 ≈ 0.67, below the strict 0.7 threshold (§3.1 no-silent-merges), so each line mints a template; the 4-token siblings of the same family (0.75) merge fine. The failure-flag confound turned out to be a red herring. #444 settled how to handle the fragmentation (2026-07-10, maintainer-approved): of the three options — tokenizer masking, length-aware thresholding, and accept-and-scope-C2-per-service — option 3 shipped (the per-service gate, RFC 0006 §3.4.3, PR #451); masking is parked as a future strategic RFC (no commitment; a Collector transform or redaction processor can polish high-cardinality infra tokens upstream) and length-aware thresholding was rejected. The safety story held throughout (bounded memory per RFC 0023, per-service C1 perfect).

The per-service decomposition is now the first-class bench gate (ourios-bench --gates c2 prints it whenever any service bucket exists — distinct service.name values plus any <unknown>/<other>, so a single-service or plain-text corpus shows its one gated row too); template creation is a globally-monotonic event attributed to the minting service, so per-service creations partition the whole-corpus count exactly (2 + 17 + 1 + 3 + 14,608 = 14,631) in O(services) memory — no per-service id set. As of #444 (option 3) this decomposition is the gate: C2 is evaluated per service and folds over the services that clear the 1 M-line floor, with the whole-corpus ratio kept as a diagnostic (RFC 0006 §3.4.3).

What the fragmentation actually costs — B2 pricing (indicative, local M-series). Running the B2 windowed query on the fragmented (kafka) vs. converged (cart) service isolates the impact:

The deployed time/column pruning floor is identical whether a service has 2 templates or 14,608 — a 1 h window prunes 48 of 49 row groups either way (reconfirming the RFC 0023 graceful-degradation result on a fresh corpus). Fragmentation does not cost query latency or pruning. What it costs is template-exact query precision: probing cart’s dominant template (id 1 in this run — a run-specific identifier, not a canonical one) recovers 1.78 M / 2.76 M rows (one template is most of the corpus) but only 11,523 / 136,790 on kafka, because kafka’s dominant event is scattered across ~11,651 ids — a single template_id probe recovers only that one id’s slice (11,523 rows), not the full dominant event. So the fragmentation is a query-capability / thesis-value tradeoff, not a performance one; the pruning path degrades to the first-class-column floor unharmed. #444 accepted that tradeoff on hostile infra logs: the per-service gate makes C2 acceptance honest without masking, and any future masking is deferred to an upstream Collector processor or a dedicated RFC.

9.13 Results — 2026-07-12 (indicative, ci-runner) — RFC 0031 comparative program vs Grafana Loki (runs #8–#18)

Purpose. The first recorded numbers for the RFC 0031 comparative program — Ourios against Grafana Loki, the incumbent CLAUDE.md §1 defines the project against. These are the §7 calibration inputs the RFC’s open questions ask for, not gate verdicts: the L-gate margins are the RFC’s proposed values (M_L1..M_L4 = 10, F_L6 = 3, wired as ComparativeMargins::default()), the §5 gate scenarios (RFC0031.2–.11) are still red stubs, and the harness reports each pair under its provisional margin rather than asserting it. Every “PASS”/“fail” below is provisional pending the §7 freeze — a maintainer step; the open inputs are enumerated in point (4) of the closing Assessment.

Corpus. corpus/otel-demo-v8 (the §9.12 capture): 4,948,596 log records, 2.96 GB uncompressed — the RFC 0031 §3.3 headline corpus (real OTLP, failure flags active, kafka fragmentation and all). Both systems ingest the identical OTLP stream; an OTLP partialSuccess in any push response fails the run, so neither side can silently drop lines. Reference system. grafana/loki:3.5.3, digest-pinned (sha256:3165cecce301ce5b9b6e3530284b080934a05cd5cafac3d3d82edcb887b45ecd), single-binary mode, fed over its native OTLP endpoint. Flag deviations from stock are documented below — all ingest-replay accommodations, all in Loki’s favour, per the §3.7 anti-strawman commitment. Hardware. ci-runner — indicative, not the §1 baseline; the authoritative baseline-8vcpu-32gib run remains a maintainer opt-in per RFC 0031 §3.2. Bytes-read, the primary channel, is CPU-insensitive by construction, but nothing here is quoted as authoritative. Runs. comparative-bench.yml dispatch runs (curated by hand as ever — no workflow writes §9), each with one harness delta under test. Counted runs are equivalence-gated passes over the full corpus; the two diagnostic failures (#11/#13) are listed with exactly what they carry:

In every counted run, RFC0031.1 result-set equivalence held on every pair: the two systems’ answers, keyed (timestamp_unix_nanos, body_bytes), were multiset-identical at 4.9 M-record scale. Runs #11/#13 were L3-flicker diagnostics (an ingester-visibility artifact, fixed in #490 — see the deviations list); their table rows above note exactly what each carries. Every dispatched run appears in the table, and the per-class tables below carry a row for every run in each quoted streak (L1: #15/#16/#17; L3: #14/#15/#16/#17), so the streaks audit from this entry alone.

The metric (§3.6 as amended 2026-07-12). The Ourios figure is the total bytes fetched from object storage per query: count scan + row materialization + template-registry derivation. Loki is reported on two channels: storage-side (query-stats compressedBytes + headChunkBytes — the conservative apples-to-apples counterpart of Ourios’s fetched compressed-Parquet bytes; the harness evaluates gates primarily on this) and totalBytesProcessed (decompressed engine-side work, which overstates Loki’s storage reads by the chunk compression ratio; reported as context). Which channel the frozen §7 gates ride is an open maintainer decision.

Program history — the biased ruler, retired. Runs #5–#7 predate the §3.6 measurement-fidelity amendment and measured the Ourios side as the count scan alone (e.g. run #7’s severity figure of 609,498 B and its “146.9×”-style ratios), silently excluding the row-materialization and registry IO while Loki’s counterpart figure includes delivering results. Those runs are program history only and are not citable; every number below is on the honest total.

L1 — template-exact lookup (must-win, the flagship class): provisional PASS, widest margins. Pair: template_id == 4323 (2 rows) vs the LogQL line-filter needle "Updated connection-accept-rate max connection creation rate to" over every stream — the picker proves the two select identical row sets before the pair counts. Loki has no template concept, so its honest equivalent is a substring scan of the whole corpus; Ourios rides the writer’s existing bloom filter on template_id.

Above the provisional M_L1 = 10 on both channels, in every run since the pair landed (third consecutive pass at #17). The Loki side is structural: no template id → nothing to prune with.

L3 — trace correlation (must-win, OTLP-native): provisional PASS after blooms. Pair: every log line for one trace_id (9 rows). trace_id is high-cardinality by construction, so it cannot be a Loki label (§3.3’s machine-checked disallowlist); Loki’s honest query is a structured-metadata filter over all streams.

Run #12 is the honest before-picture: without blooms Ourios itself had to fetch the trace_id column corpus-wide, and the storage-side ratio (1.41×) was nowhere near the margin. The blooms (implemented in #489; the RFC 0005 §3.6 amendment recording them, with this as its measured evidence, is #491) collapse the fetch 15×, and the pair has now passed the provisional margin on both channels three runs in a row. As with L1, Loki’s side is structural: a trace cannot be pre-narrowed to a label stream, so it scans and decompresses everything in the window.

L2 — severity predicate (must-win family): parity-plus storage-side, ~33× processed — not a provisional 10× pass. Pair: lowest-volume single-severity_text band on the highest-volume service, full corpus span, 1 row. The run series doubles as the read-path optimisation ledger (component split: count scan + materialize + registry):

(Run #8’s Loki side: 2,880,784 storage / 89,184,711 processed.) Across the later reproductions the storage-side ratio sits at 1.05–1.31× and processed at ~33–39×, the spread being entirely Loki-side wobble (below). Reading: on the honest metric Ourios went from losing the storage channel (0.67×) to parity-plus via two read-path fixes, and wins decisively on engine work — but this is not a 10× storage-side pass, and no amount of wobble makes it one. The remaining named levers: the constant 513,862 B template-registry derivation, 20–29 % of every small-answer query’s total (the RFC 0033 cached-template-map candidate), and write-side page/row-group sizing.

Time-window browses (L6 floor family): published loss on the storage channel. Pairs: all lines of the highest-volume service in a clean k-row window (the promoted-column bloom’s worst case), plus run #17’s diagnostic — the same shape scoped to the lowest-volume service (“ad”, ~34 s window), where the service.name bloom could in principle skip. Floor gate as reported here: a bytes-read floor analog (Ourios ≤ 3× Loki, i.e. ratio ≥ 0.33) — the harness applies the §7 F_L6 factor to this entry’s bytes channels. Note the §5 gate as written (RFC0031.7) defines the L6 floor on latency p50 — measured in run #18 (see the latency section below), where the gate as written passes on all three window pairs; the bytes framing here remains the conservative reporting channel pending the §7 freeze.

This is the honest loss the RFC’s L6 disposition anticipated, and it is published as §5 RFC0031.11 demands: on a browse-k-rows query Loki reads only the tiny chunk slice its label stream + time index point at, while Ourios pays fixed per-query costs (the registry constant plus row-group-granularity materialization) that dwarf a k-row answer. The #486 late-materialization fix halved the loss and lifted k=2000 past the processed floor; storage-side stays 0.007–0.018 vs the 0.33 floor on current code. Run #17’s diagnostic sharpens the why: scoping to a low-volume service improves Ourios only ~22 % and flips the processed floor to pass, but there is no bloom collapse — v8’s hour partitions each hold roughly one row group containing all services, so the promoted service.name bloom has nothing to skip. The tier-changing lever is write-side layout (service clustering / row-group sizing — hazard #4 territory, an RFC-level change), not query-side tuning.

Latency (§3.6 channel, run #18 — the program’s first). Median of 7 warm repetitions per pair per system, measured only on correctness-verified pairs; Ourios timed in-process, Loki over localhost HTTP (negligible at these magnitudes; stated because latency is corroborating, not sole-gating):

Two findings this channel settles. First, the young-engine latency risk the RFC hedged against (“a latency loss + bytes-read win = sound architecture, young implementation”) did not materialize: Ourios answers every pair in 39–86 ms — a flat, fixed-cost-shaped profile — while Loki spans 13.8 ms to 24.1 s, and on the needle classes the wall-clock gap is interactive-vs-batch (75 ms vs 23–24 seconds). Second, scenario RFC0031.7 evaluated as written — on latency — PASSES on all three window pairs (0.34, 3.43, 1.32, all ≥ 1/3 at F_L6 = 3), and Ourios is outright faster on two of the three; the storage-channel loss published above is real as a bytes statement, but the RFC’s own L6 gate holds the floor. Which channel the frozen L6 gate uses is part of the §7 decision.

Determinism note. For repeated measurements of the same build and configuration, Ourios’s bytes are byte-identical (the store build is deterministic) — differences between runs are exactly the harness/optimisation deltas the table names, which is what lets the run series read as an optimisation ledger. Loki’s storage-side figure wobbles run to run (severity pair: 2.67–3.35 MB) with chunk boundaries and flush timing; ratios quoted against Loki carry that band.

Documented Loki flag deviations (all in Loki’s favour, per §3.7). The committed harness starts Loki with, and comments, exactly these deviations from stock:

• -validation.reject-old-samples=false — the frozen corpus is weeks old; stock Loki would reject the replay outright.
• -querier.query-ingesters-within=0 — stock Loki (default 3 h) skips ingesters for queries over weeks-old ranges, making rows still in unflushed low-volume chunks invisible (the run #11/#13 L3 flicker; diagnosed via ingester.totalReached: 0, fixed in #490). Disabling the cutoff means ingesters are always consulted — without it Loki’s answer to an old-range query is silently incomplete.
• Raised ingestion + per-stream rate limits (-distributor.ingestion-rate-limit-mb=512, -distributor.ingestion-burst-size-mb=1024, -ingester.per-stream-rate-limit=512MB, -ingester.per-stream-rate-limit-burst=1GB) — replay is far faster than the capture’s real-time rate.
• Raised internal gRPC message caps (-server.grpc-max-recv-msg-size-bytes=16777216, -server.grpc-max-send-msg-size-bytes=16777216) — runs #2–#4 failed on the same ~5.27 MB internal message regardless of our outer batch size: a single kafka-service LogsData line’s content alone inflates past Loki’s stock 4 MiB internal cap. Raising it (standard operator tuning) lets Loki accept the data at all, preserving the identical-ingest precondition the equivalence check requires.

Assessment. (1) The two classes the thesis stakes itself on hardest — L1 template lookup and L3 trace correlation — pass their provisional must-win margins on both channels, reproduced across three consecutive runs, and in both cases Loki’s cost is structural rather than tuning: no template concept, and no way to index a trace id. (2) L2 is parity-plus on storage and a ~33× processed win, honestly short of a 10× storage claim, with two named levers still on the table. (3) The window browses are a published storage-channel loss whose mechanism is understood (fixed per-query costs vs v8’s one-row-group-per-hour layout); the lever is write-side and RFC-sized. (4) Nothing here is frozen: the §7 inputs — the primary metric channel (storage-side vs processed), the must-win margins and floor factors, and whether the time-window pairs reclassify from gated floor to diagnostic — are open maintainer decisions, and this entry is the calibration evidence for them, not their resolution.

9.14 Results — 2026-07-13 (indicative, ci-runner) — comparative run #20: frozen gates on main, RFC 0033 acquisition

First dispatch on main after the §7 partial freeze and after the RFC 0033 cached template map merged (#511–#513). Job: run #20 (29255000054), exit 0.

Frozen gates. All asserting gates pass on main — M_L1/M_L3 storage margins and the F_L6 latency floors held; equivalence held on every pair. The dispatch is functioning as the regression gate the freeze intended (run #19 proved it on the branch; this run proves it on main).

RFC 0033 acquisition (the run’s purpose). Every pair reports:

template-map acquisition (RFC 0033): cold (audit fold, 513862 B; no artifact published)

• The registry component is byte-identical to run #8’s baseline (513,862 B constant per body-rendering query): the cache regressed nothing, exactly as the advisory design promised.
• But the write-through never published on this corpus, so no pair ever ran warm and the RFC0033.6 corpus gate (warm/cold ≤ 1/10) could not be measured.
• The explanation consistent with the run’s outputs is §3.2’s size abstention: the artifact is uncompressed JSON carrying every (template_id, version) canonical template string, while the 513,862 B it must undercut is zstd-compressed Parquet of the same strings (plus their event history). On v8’s template set the JSON evidently meets or exceeds the fold, and the guard refuses a publish that would make warm acquisition cost more bytes than the fold it replaces. (A publish IO failure would leave the same “no artifact” label; the §3.7 publish-outcome telemetry distinguishes the two in a served process, but the bench harness does not export metrics — the amendment run should print the outcome explicitly.)

Consequences recorded.

1. RFC 0033 status reverted green → red (this PR): RFC0033.6’s corpus arm is undischarged. The local-shape arm (55.8× on the 64-event fixture) stands.
2. M_L2 stays frozen-deferred — §7’s unfreeze condition (the RFC 0033 warm measurement on the headline corpus) was not met.
3. The lever is an artifact encoding amendment (format_version 2, compressed body). The same template strings zstd-compress into the 513,862 B audit Parquet with full event history alongside, so a compressed artifact is expected to land well below the fold size — to be measured, not assumed. Abstention semantics stay: publish only when the artifact beats the fold.

9.15 Results — 2026-07-14 (indicative, ci-runner) — comparative run #21: the v2 compressed artifact publishes and runs warm

Dispatched from the RFC 0033 v2 implementation branch (PR #522, the measure-before-merge step). Run 29343438434.

The RFC 0033 answer. The zstd artifact published on the corpus (no abstention — the run #20 ambiguity is resolved by the new per-pair outcome labels), and every measured pair ran warm:

template-map acquisition (RFC 0033): warm (one artifact GET, 187904 B compressed)

• warm = 187,904 B (the compressed artifact, GET cost) vs cold = 513,862 B (the audit fold, byte-identical to run #8) — warm/cold ≈ 1/2.73, a ~326 KB cut off every body-rendering query’s honest total.
• The original RFC0033.6 ratio gate (≤ 1/10) does not pass on this corpus: the artifact is O(live template state), the fold is O(audit history), and otel-demo-v8 is young — the amended gate (≤ 1/2, dated 2026-07-14 in the RFC) asserts the real margin and ages upward. See the §5.6 amendment for the full argument.

The test failure is not an Ourios finding. The run exited 1 on one pair: loki returned 0 of 9 expected rows for [trace correlation, L3] before timeout — the Loki-side low-volume-chunk race (the run #12-era flicker), resurfacing on the shared runner despite the #490 flag fixes. All other pairs measured; the report and every RFC 0033 number printed before the panic. A rerun for a clean L3 pair is queued as run #22.

9.16 Results — 2026-07-14 (indicative, ci-runner) — runs #22 and #23: the v2 artifact asserting, M_L2 unfrozen

Two dispatches after the RFC 0033 v2 merge (#522):

• Run #22 (29352282162, from main): exit 0 — the clean-record run. All then-frozen gates passed, the L3 pair measured cleanly (run #21’s Loki-side flake did not recur), and every pair ran warm on the compressed artifact.
• Run #23 (29353634499, from the M_L2-unfreeze branch): exit 0 — the first run with the full assertion set live. L2 processed (PRIMARY, frozen 10) 43.97×; L2 storage-side floor (frozen 11/10) 1.49×; L1 storage 108.3× and L3 storage 24.9× against their frozen 10s; latency floors held; and the RFC 0033 §5.6 acquisition gate asserted warm = 187,905 B compressed on every pair against the 513,862 B fold (ratio ≈ 1/2.73, gate ≤ 1/2).

With #528 merged, §7’s measurable gates (M_L1, M_L2, M_L3, F_L6) are all enforcing on every comparative dispatch; M_L4/F_L7 stay deferred until measured. RFC 0033’s §5 is fully discharged: the corpus arm passed as measured (#21), and passed again as an asserting gate (#23) — the status flips red → green with this record.

9.17 Results — 2026-07-17 (indicative, ci-runner) — L4 frequency aggregation measured (PR #536 arc)

The last unmeasured must-win class. The L4 workstream’s own dispatch sequence (~23 real comparative-bench runs across the arc — a numbering distinct from §9.16’s) fixed three genuine harness bugs early (LogQL escaping, a control-flow ordering bug, a missing picker row ceiling), then spent the balance of the runs on a persistent completeness shortfall that no harness-side fix closed: Loki never returned 100% of any L4 candidate’s expected rows on this corpus. Every mechanism checkable from the harness side was ruled out directly — exact (timestamp, body) ingester dedup (corpus analysis found zero collisions), push-path drops (partial_success asserted clean on every push), Loki’s own warn/error logs (silent), and its loki_discarded_samples_total accounting (zero, of any kind). The residual matches open upstream grafana/loki#10658 (wide-time-range queries silently missing a small percentage of lines, no maintainer-identified root cause). RFC 0031 §7 records the resulting amendment: L4_COMPLETENESS_MARGIN = 0.90, checked per group_key with phantom-cell and per-key-overcount hard-fails — the full five-iteration comparator design trail lives there.

The measured pair (picker floors L4_MAX_ROWS = 100_000, L4_MIN_AVG_INTERVAL_SECONDS = 100 — lower-frequency candidates measure more completely; mechanism uncharacterized, NOT dedup): template_id=60 (Periodic task <type> generated), param(0), bucket(12h), 1,197 expected rows, group cardinality 4.

Ourios’s side is constant at 47,995,205 B total (the honest §3.6 metric). Four consecutive equivalence-verified measurements in a 3.69–3.73× / 86.5–87.1× band: the shape mirrors L2 pre-freeze — a strong processed-channel win with storage closer to parity. M_L4 stays §7-deferred (both channels reported, nothing asserted); the proposed freeze shape on #498 is the L2 precedent — processed-channel must-win at 10× plus a storage-side floor (L2’s is frozen at 1.1×; L4’s measured 3.69–3.73× storage band would clear a similar floor with real headroom).

Follow-on hardening, so the 2 h dispatch confirms rather than discovers (#538/#499, closed via #539–#542): a proptest property suite over the margin comparator (its properties were verified by manually re-introducing the three historical comparator bugs and confirming each is caught — the evidence trail is PR #539’s record, not a standing mutation-testing harness), per-pair completeness recorded as a machine-readable artifact on every dispatch, a backdated wide-time-range arm in the per-PR loki-interop job running the dispatch’s exact Loki flags (one shared constant — config drift between the 1-minute test and the 2 h run is now unrepresentable), and a dispatch class filter for targeted re-runs.

9.18 Results — 2026-07-18 (indicative, ci-runner) — the M_L4 freeze’s first asserting run

Workflow run 29659514874, dispatched from the freeze branch (PR #548) with the L4 gates enforcing for the first time — M_L4 = 10 on the processed channel (primary) plus the 1.1× storage-side floor (m_l4_storage_floor_tenths = 11), the L2 shape per the §7 decision (maintainer, 2026-07-18). Exit 0; every frozen gate Decided { pass: true }:

L4 completeness this run: 1151/1197 = 96.2% — inside the §7 margin and the §9.17 band (95.3–97.5%). (Latency-floor advantages are oriented loki_p50/ourios_p50 — above 1 means Ourios faster; the floor passes at ≥ 1/3.) With this run the dispatch asserts every §7 value except F_L7 (deferred until L7 is first measured): an L4 band degradation below either frozen value now fails the run instead of printing a smaller ratio.

9.19 Results — 2026-07-20 (indicative, ci-runner) — first D1/D2 sustained-ingest soak (#558)

Purpose. The first run of the RFC 0009 D1/D2 soak harness (#558, ourios-bench soak + soak-bench.yml): the full in-process pipeline — IngestPipeline group commit (100 ms WAL batch window) → seal → sweep → compact — under one hour of paced synthetic OTLP load. Record timestamps ride a ×60 synthetic clock and the same synthetic “now” feeds the sweeps, so hour-sealing exercises continuously; D1’s ack latencies are wall-clock. Workflow run 29717165102; the JSON report is the run’s artifact. Hardware: ci-runner (4 vCPU) — indicative, not the §1 baseline.

Numbers (release build, defaults: total offered load 100,000 lines/s — the paced target, not D1’s per-core bar — batch 1,000, 4 workers, 10 s sampling):

Reading. D2 is a clean pass: compaction kept pace with a full hour at target with a backlog that never exceeded one partition. D1 splits: the machine sustained the 100k lines/s target with ack p99 inside the bar, but normalized per core (÷4 workers on 4 vCPU) it lands at ~25k lines/s/core against the ≥ 100k/core bar. Open reading for the maintainer: whether the bar means single-core-scaled throughput (then this is a real 4× efficiency gap to close) or per-node throughput on baseline hardware (then the authoritative baseline-8vcpu-32gib run decides). The commit path’s fsync overlap needs ≥ 2 threads, so a literal 1-worker measurement under-credits by construction. Either way the harness now measures instead of guessing, and the number is honest: no bar was reworded to fit the result.

9.20 Results — 2026-07-20 (authoritative, baseline-8vcpu-32gib) — D1 capacity probes: the single-tenant ceiling

Purpose. §9.19 left D1’s per-core bar as an open reading and noted the run had paced at the target rather than probing capacity. These are the first capacity probes, on the §1 baseline class (8 dedicated vCPU / 32 GiB) — baseline runs are maintainer opt-in per RFC 0031 §3.2, and this one was. Ad-hoc VM run (not a workflow dispatch): ourios-bench soak at 0979d14, release build; per-probe JSON reports retained by the maintainer alongside this record’s source run log.

Ladder (10-minute probes, 8 workers, one tenant, ×60 synthetic clock):

The identical ~86k plateau at every offered rate, with ack latency pinned at ~6 s, is a saturated pipeline: at saturation the harness’s in-flight bound (a 512-batch semaphore in the load loop, #558) sets the latency (512 batches × 1,000 lines ÷ 86k lines/s ≈ 6 s) while the plateau itself is the pipeline’s service rate — raising the permit count would lengthen the queue, not the throughput, because the per-tenant miner hand-off is in-order and sequential by design (CLAUDE.md §3.7 per-tenant trees; the least-common-mechanism choice’s deliberate flip side). Stated carefully: ≈ 86k lines/s is the observed single-tenant ceiling under this probe configuration (batch 1,000, 8 workers, this corpus shape), consistent with the sequential-miner explanation; a different batch size or record mix could move the number some, but not onto a cores axis.

Tenant-parallel check (8 concurrent single-tenant soak processes, 2 workers each, 5 minutes at offered 100k each — an approximation: separate WALs/stores per process): 41,895–43,335 lines/s each, ≈ 341k lines/s aggregate — ~4× the single-tenant ceiling on the same box. Node ingest capacity scales with tenant parallelism, not core count.

Reading. The strict per-core bar (≥ 100 000 lines/s/core) is not merely uncalibrated — for single-tenant load it measures a dimension the architecture deliberately does not scale on. D1’s falsifier (“a meaningful share of production traffic per node”) is a per-node, multi-tenant statement: approximated here at ≈ 341k lines/s (≈ 29 B lines/day) — a multi-process stand-in, not yet a per-node measurement (separate WALs/stores neither share a commit stream nor contend on one store); the honest in-process --tenants N measurement is tracked in #567. D2, by contrast, passed at full saturation on every probe. The bar recalibration (per-node multi-tenant must-win + per-tenant ceiling as recorded diagnostic, the RFC 0011 must-win/diagnostic precedent) is a pending maintainer decision; until it lands, D1 stays FAIL-as-written and this record is the evidence, not the verdict.

9.21 Results — 2026-07-20 (authoritative, baseline-8vcpu-32gib) — in-process multi-tenant ceiling + the serialization profile

Purpose. The §9.20 tenant-parallel figure (≈ 341k lines/s) was a multi-process approximation flagged as such; #567’s in-process --tenants N mode (merged #570) is the honest instrument. Ad-hoc VM runs at 9ad3158-era code; the per-run JSONs and the profile artifacts are retained by the maintainer outside the repository (the gitignored local scratch/ tree), as with every ad-hoc VM record in this series.

Finding 1 — node capacity is FLAT across tenants. 10-minute saturating soaks, 8 workers, one shared WAL/commit stream: 1 tenant ≈ 86.0k, 8 tenants 85.8–86.2k (three offered rates), 16 tenants 87.3k lines/s. The multi-process 341k was ~4× optimistic precisely because separate processes had separate commit streams.

Finding 2 — the ceiling is software serialization, not hardware. Profile at saturation (flamegraph + per-thread pidstat): 1.2 of 8 cores busy — ~85% idle; no thread above ~33%. Root cause (crates/ourios-ingester/src/receiver/pipeline.rs:314–354 at that commit): the global WAL-seq gate + the global miner mutex serialize all tenants, with the miner match AND the sink emit (including size-triggered Parquet encode + store put — I/O) inside the single-file section. Issue #571; design → RFC 0035 (specified). D2 passed at full saturation in every run.

9.22 Results — 2026-07-20 (authoritative, baseline-8vcpu-32gib) — RFC 0035 Design A prototype A/B

Purpose. The RFC 0035 §6 pre-implementation measurement: main (9ad3158 lineage) vs the Design A prototype (rfc0035-prototype 987b781 — ordered mining under the gate, sink emit + triggered publish moved to a bounded concurrent pool, crude quiesce barrier). Same VM, back-to-back 10-minute saturating soaks (8 tenants, offered 800k, 8 workers). Artifacts retained by the maintainer outside the repository (gitignored local scratch/ tree).

Design A multiple on the baseline class: 1.61× (implied residual serial fraction ≈ 0.62 — the ordered miner phase + WAL group commit). Honest note: this is below the M-series indicative 1.82× — the prediction that the baseline multiple would land higher was wrong; the EPYC’s slower single-thread makes the still-serial ordered phase relatively costlier. Saturation ack latencies are queue-bound in both arms (the in-flight bound), per §9.20’s reading — the p99 bar applies at sustained rates below capacity. This 132k lines/s/node figure is the §6 input to RFC 0035’s target and to the RFC 0034 D1 recalibration; whether Design B (§4) is ever escalated is judged against the recalibrated bar, not the old per-core one.

9.23 Results — 2026-07-21 (authoritative, baseline-8vcpu-32gib) — the RFC0034.2 / RFC0035.4 asserting soak

Purpose. The asserting run for the recast D1 bar: RFC 0034’s RFC0034.2 (per-node must-win with the observable below-saturation condition) and RFC 0035’s RFC0035.4 (the serialization is actually relaxed) in one measurement — the RFC 0034 §7 one-run-two-records question resolved as one run, one record, cited by both. Ad-hoc VM run at main d2c622e (the RFC 0035 production implementation, #577, plus its review fixes, #579), release build; the JSON report is retained by the maintainer outside the repository (gitignored local scratch/ tree), as with every ad-hoc VM record in this series.

Shape (the RFC 0034 §3.1 asserting shape): one-hour soak, soak --tenants 8 (N = cores), offered exactly 100,000 lines/s — the bar rate, paced, not a saturating probe — batch 1,000, 8 workers, ×60 synthetic clock, 30 s sampling.

Reading. The recast D1 must-win asserts for the first time and passes: 0.08% pacing loss at the bar rate is achieved ≈ offered, which is the below-saturation proof, and the p99 (153.63 ms) is measured over that same run, inside the 200 ms bar. The p50 rides the 100 ms group-commit window (the CLAUDE.md §3.4 batch default) — ack latency at this rate is floored by batched fsync, not queue-bound. The 723.70 ms max is a tail spike outside the bar’s percentile; the bar is p99, and it holds. One honesty note: the harness’s printed per-core “FAIL” line is the pre-RFC-0034 bar mechanically applied — per RFC 0034 that number (12,490 lines/s/core) is now a recorded diagnostic, and the recalibrated per-node must-win is what judges the run. This run satisfies RFC0034.2 and RFC0035.4 simultaneously: the production Design A sustains the recast bar with margin to spare over the pre-RFC ~82k saturation baseline (§9.22), with D2 passing over a full hour of sustained multi-tenant load.

9.24 Results — 2026-07-21 (authoritative, baseline-8vcpu-32gib) — first authoritative comparative run: all frozen gates pass

Purpose. The RFC 0031 comparative program’s move from indicative to authoritative — the last open decision on the #498 scoreboard. Every number in the §9.13–§9.18 series was measured on ci-runner and labelled indicative; per RFC 0031 §3.2 the baseline-hardware run is a maintainer opt-in, and this one was. Ad-hoc VM run on the §1 baseline class (8 dedicated vCPU / 32 GiB), not a workflow dispatch, but an exact replica of the comparative-bench.yml dispatch recipe at main 9deecb1: frozen corpus/otel-demo-v8 (4,948,596 records), grafana/loki:3.5.3 digest-pinned via Docker, OURIOS_COMPARATIVE_CLASSES=all, release build, every §7 FROZEN gate asserting. Run logs and the machine-readable comparative-results.json artifacts for both runs are retained by the maintainer outside the repository (the gitignored local scratch/ tree), as with every ad-hoc VM record in this series.

Two runs, recorded honestly. Run 1 (5,798 s) measured every pair and passed every gate it printed — L2 39.51× processed / 1.275× storage floor, L1 97.56× storage, L4 84.93× processed / 3.59× storage floor, both L6 latency floors — but failed overall: the L3 pair hit the known transient Loki-visibility flicker (loki returned 0 of 9 expected rows … before timeout) — the class addressed by the #490 flag fixes, previously hit in runs #11/#13/#21 — so the harness hard-failed the run rather than salvage it. The decision rule applied: one deliberate, configuration-identical retry; had the flicker recurred, the next step would have been engineering, not rerolling. It did not recur. Run 2 (5,556 s) is the counted authoritative run: exit 0, 1 passed / 0 failed, all 11 frozen gate decisions Decided { pass: true }, L3 measured cleanly, RFC 0033 acquisition warm on every body-rendering pair (one artifact GET, 187,906 B compressed), equivalence held on every pair, and L4 completeness 1147/1197 = 95.8% — inside the §7 0.90 margin and the §9.17 band (95.3–97.5%).

Run 2 — the counted numbers (honest §3.6 totals; Loki bytes per channel; latency oriented loki_p50/ourios_p50, where > 1 = Ourios faster):

Reading 1 — the frozen-gate program confirms at authoritative class. Every §7 FROZEN value asserted and passed on the baseline hardware with no recalibration. On the bytes channels the margins are equal or wider than the ci-runner series: L1 storage 97.82× vs the §9.13 calibration streak’s 77.2–77.7× (and in the neighbourhood of the later asserting runs’ 102–108×); L3 22.52× vs 21.2–21.9×; L2 storage floor 1.239× inside its 1.05–1.49× history. L4’s 3.60× storage floor sits a whisker below the §9.17 indicative band (3.69–3.73×) — Loki-side storage wobble scale (§9.13’s determinism note), and more than 3× the frozen 1.1× floor either way; L4 processed (85.14×) is inside its 84.9–87.1× history. Bytes-read being CPU-insensitive by construction (§9.13), agreement here is expected — but now it is measured, and the thesis’s must-win claims are no longer resting on indicative hardware alone.

Reading 2 — the needle latency ratios compress on dedicated hardware, exactly as caveated. L1 90.44× and L3 77.99× against the ci-runner indicative 308×/323×: Loki’s corpus-wide needle scans drop from 23–24 s to ~10 s on the faster box while Ourios stays in its flat fixed-cost profile (57–129 ms warm p50 across every pair). This is §9.13’s / #498’s “a tuned environment compresses latency ratios; the structural asymmetries remain” caveat, now measured rather than hedged: the wall-clock gap on the needle classes is still interactive-vs-batch (~110–129 ms vs ~10 s), the L6 floors still hold (0.37 / 4.34 / 0.76 diagnostic), and the bytes asymmetry — the primary channel — did not compress. Quote the latency channel from this entry, not the indicative one.

Reading 3 — post-RFC-0035 code, gates held without recalibration. 9deecb1 carries the RFC 0035 Design A ingest restructuring (#577/#581). Every read-path byte count and every frozen gate held with no adjustment, consistent with RFC0035.5’s guarantee that the concurrency change alters no on-disk artifact.

With this record the #498 authoritative-rerun checkbox — the last open decision on that scoreboard — is discharged. The remaining storage-side lever named in §9.13 (write-side layout, hazard #4) stays parked on its own line, unchanged by this run.

9.25 Results — 2026-07-22 (indicative, local M-series) — D2 / D3 after the RFC 0036 sorted compaction

Purpose. RFC 0036 slice B (RFC0036.3) makes compaction sort the partition by (promoted service.name, time_unix_nano) via the §3.2 external merge sort and rotate compacted row groups at the smaller compacted threshold (the fixed 32 MiB in effect at this run; later adaptive — §9.30). §5 requires that D2/D3 — RFC 0009’s compaction-throughput and file-band properties — survive the sort. This is the “first measurement” §7 asks for, at red: the band from which the sorted D2 is set. Indicative, not authoritative — the baseline-VM sorted-vs-unsorted rerun is deferred to validated (a paid, maintainer-opted run, per RFC 0031 §3.2 / the “bench on ci-runner first” discipline).

Hardware / run. Local M-series developer machine (P-cores; taskpolicy -B to escape the Claude-Code background-QoS E-core throttle), release build at the slice-B head. The ourios-bench compaction bench, band-scale one-shot mode (OURIOS_COMPACTION_BASELINE=1 FILES=32 ROWS=4800 BODY_BYTES=4096 — the exact §9.7 shape, ~453 MiB of input in one partition), plus the criterion micro-sweep to confirm the groups still execute with the sort in the path.

Reading — “sorting is not free,” measured honestly. Sorted D2 on this box is ~138 MiB/s against §9.7’s 166.8 MiB/s unsorted. The two numbers are not the same hardware (local M-series vs the 8 vCPU EPYC-Milan baseline), so this is not a clean sorting-overhead delta — it is a fresh indicative measurement of the sorted path, plus the observation that adding a full external-merge-sort pass (decode → per-input stable sort → spill/merge, or the in-memory skip-spill short-circuit) to the rewrite lands in the same order of magnitude as the prior unsorted copy-through and comfortably clears “keeps up.” The clean same-hardware sorted-vs-unsorted delta is a validated baseline-rerun item; it is deliberately not asserted here. The D2 band is set from this measurement (indicative floor ~120 MiB/s on this class) and is not an in-repo wall-clock gate — wall-clock gates flake, so the in-repo RFC0036.3 assertion is structural (D3 file band + the §3.2 memory bound); the throughput lives here in §9 and in the bench (RFC 0036 §6).

Reading — D3 holds unchanged. The headline: the sort did not touch the file band. Compaction still emits exactly one file per partition, 452.7 MiB, squarely in the H4 256 MiB–2 GiB target with zero sub-128 MiB files — the RFC 0036 §3.3 amendment drops the compacted row-group threshold (32 MiB, ~14 groups here) but leaves the file band untouched, and D3 measures files. The structural side (one output file, small-file count → 1, rows conserved, sorted layout declared) is pinned deterministically in ourios-parquet’s rfc0036_3_compaction_properties_preserved (tests/it) and the merge proptests; the §3.2 peak-memory bound (forced-spill residency = one input + F×batch, never whole-partition) in rfc0036_3_forced_spill_peak_far_below_whole_partition. These wall-clock figures are the indicative stamp; the authoritative sorted D2/D3 awaits the validated baseline rerun (RFC0036.5’s frozen-gate pass will accompany it).

9.26 Results — 2026-07-22 (authoritative, baseline-8vcpu-32gib) — RFC 0036 comparative rerun: no regression + the single-file-harness finding

Purpose. RFC0036.5’s deferred validated item: rerun the frozen RFC 0031 comparative dispatch at post-RFC-0036 main (HEAD 5e5aa66) on baseline hardware, and confirm the frozen gates still pass on the sorted-compaction code. The v8 corpus (frozen corpus/otel-demo-v8 release), one Loki container + one replay, same methodology as §9.24.

Hardware / run. Fresh ccx33 (8 dedicated vCPU EPYC-Milan / 32 GiB = baseline-8vcpu-32gib), release build, Docker for the Loki testcontainer, box deleted on exit. ~75 min/attempt.

Result — every measured frozen gate passes, in §9.24’s band (no regression):

The finding — RFC 0036 is not exercised by this harness, and the run proves it. Every ourios_bytes_read came back byte-for-byte identical to §9.24 (L1 1,032,727; L2 2,223,173; L4 47,995,205; L6 k=100 1,931,911; L6 k=2000 4,202,473). build_comparative_store writes exactly one ingest file per partition (append_record keeps one Writer per partition), so compact_partition no-ops (it needs ≥ 2 files) and RFC 0036’s compaction-time sort never runs. The comparative store is therefore identical pre/post-0036 — which is why the gates pass unchanged (no regression) and why this harness cannot measure RFC 0036’s window-materialization win. Making it accrue multiple files per partition then compact would re-base RFC 0031’s frozen-gate store; that is deferred as future harness work, not a validated blocker (the RFC0036.2 bytes channel is a §2.2 diagnostic, not a gate).

Two orthogonal noise items (neither an RFC 0036 layout regression): attempt 1 measured L3 cleanly (21.6× storage) but landed the L6 k=100 latency floor at 0.311 < 0.333 (marginal tiny-query noise; attempt 2 passed it at 0.364); attempt 2 hit the known Loki L3 flicker (0/9 rows before timeout — the §9.13 class, #490). One justified retry per the retry-once rule; no third.

9.27 Results — 2026-07-22 (in-repo, deterministic) — RFC0036.2 window-materialization before/after

Purpose. The RFC0036.2 materialization diagnostic that §9.26’s harness cannot produce, measured directly on a genuinely-compacted store. ourios-querier’s rfc0036_2_materialization_before_after (tests/it) runs the identical L6 k=100 window query (1 s at the 10 ms grid, one service) against two stores built from the same synthetic multi-service hour, and reads the materialization bytes — the compressed column chunks of the row groups a service = target ∧ time ∈ window scan cannot prune — from each file’s footer.

Reading. Identical 100-row answer from both; the sort takes the window from materialising the whole file (the single-row-group unsorted ingest parquet — no row group prunes) to a contiguous minority (2 of 6 groups) — a 1.43× materialization-bytes win. Modest by design, and honestly so: the compacted file is physically ~2× larger (six 32 MiB groups compress a little worse than one 128 MiB group — the RFC 0036 §3.3 pruning- granularity-over-bytes trade), and §2.2’s ~188 KB registry floor is exactly why the gate is the scanned-row-group bound (enforced by rfc0036_2_window_materialization_bound), not a bytes ratio. count- scan stats.bytes_read is not the materialization term — it reads only the small filter columns and points the wrong way; the footer survivor-chunk sum is the RFC §9 metric.

9.28 Results — 2026-07-22 (indicative, local M-series, synthetic-compressible) — RFC 0036 §7 compacted row-group threshold sweep (16 / 32 / 64 MiB)

Purpose. RFC 0036 §7’s first open box defers the authoritative 16/32/64 MiB compacted-threshold sweep to the paid baseline-8vcpu-32gib harness. This is the in-repo, indicative half. §9.27’s before/after used a near-incompressible random payload (chosen to cross 32 MiB with few rows) and so read the compacted file as ~2× larger on disk — a worst-case artifact of incompressible bytes, not what real logs do. Real logs are compressible and have per-service locality, and RFC 0036 sorts by service.name, which clusters similar lines. This sweep re-measures the trade on a compressible, service-clustered synthetic corpus to correct that impression and trace the actual curve. Indicative, not authoritative — local hardware, synthetic corpus; the v8-corpus L6-scanned-bytes-vs-L1/L3 sweep stays deferred to validated (RFC 0036 §7).

Hardware / run. Local M-series developer machine (P-cores; taskpolicy -B to escape the Claude-Code background-QoS E-core throttle), dev build at the sweep head. ourios-querier’s rfc0036_7_compacted_threshold_sweep (tests/it, #[ignore]d like the RFC0005.6 sizing test — not a CI gate; the gate is rfc0036_2_window_materialization_bound). It drives the shipped compaction path with the new explicit-threshold seam (compact_partition_with_flush_threshold).

Corpus. 6 promoted services, each with its own fixed log phrase (distinct vocabulary, so the §3.1 sort clusters like text) plus a handful of small varying fields — request id, user, amount, status/region enums, and 8 hex chars of per-line entropy that hold ZSTD to a realistic ratio. 2,160,000 rows (360,000/service), ~230–260 B bodies, 467,239,238 B (445.6 MiB) of raw body text, written as two interleaved ingest files then compacted. Measured compression: 7.20× (raw body ÷ the 61.85 MiB compacted file) — squarely in the realistic 5–15× band, not the ~2.4× of random ASCII.

Finding 1 — the ~2× on-disk bloat is a random-bytes artifact (the headline). On compressible, service-clustered data the compacted file does not grow as the threshold shrinks. 32 and 64 MiB produce a byte-identical file; 16 MiB is larger by only 519,823 B (+0.80%) — the cost of two extra row groups’ footer/index entries and slightly-shorter compression windows. §9.27’s “compacted file ~2× larger” was the incompressible payload amplifying that per-group overhead against near-zero codec gain; it does not generalise to real logs. The §3.1 service sort, if anything, helps compression by clustering like lines — the finer-threshold file barely moves.

Finding 2 — finer thresholds buy real pruning granularity, nearly free. The same fixed 30 s one-service window materialises 14.66 MiB at 16 MiB vs 28.94 MiB at 32/64 MiB — the finer threshold halves (1.97×) the bytes a window browse must fetch, for a +0.80% file-size cost. The answer is identical (3,000 rows) at every threshold; only the IO changed. This is the pruning-granularity-over-bytes trade RFC 0036 §3.3 exists to make, now measured on realistic data: it is a good trade, and it gets better the finer the threshold, bounded only by footer/index overhead.

Finding 3 — 32 and 64 MiB are near-identical because arrow’s default row-group row cap sets a ~30 MiB granularity floor, finer than either byte threshold. The compacted writer sets no max_row_group_size, so parquet-rs’s default 1,048,576-row cap applies. At this corpus’s ~30 encoded bytes/row that cap fills a group at ~30 MiB, which trips before the 32 MiB byte flush (and well before 64 MiB): so 32 and 64 MiB are both row-capped at ~3 groups of ~30 MiB (byte-identical files), and only 16 MiB (~590 k rows → ~15 MiB, under the cap) is genuinely byte-governed → 5 finer groups.

Correction to an earlier framing. A prior draft of this finding called the row cap a “gap” and suggested raising max_row_group_size “so the byte threshold bites.” That is backwards: raising the cap would let the 32/64 MiB byte flush govern and produce fewer, coarser groups (~2 of ~32 MiB) — the wrong direction for pruning. The row cap is a granularity floor that is currently helping. The lever that unambiguously improves window pruning is a smaller byte threshold (16 MiB halves window materialization here), the §7 authoritative-sweep question — not a larger row cap. Making group sizing byte-uniform (raising the cap) is a separate predictability choice, not a pruning win. Net: finer effective groups win the materialization trade at sub-1% disk cost, and the byte-threshold value — not the row cap — is the lever.

Conclusion. 32 MiB stays a reasonable, defensible shipped default (it already delivers the pruning mechanism the RFC0036.2 gate enforces). The compressible-data trade leans, if anywhere, toward smaller thresholds (16 MiB halves window materialization for +0.80% on disk) — the opposite of the “smaller = bloated file” worry §9.27’s random payload suggested. 16 MiB is flagged as a candidate for the authoritative v8 sweep to evaluate against L1/L3 neutrality (more, smaller row groups add footer/page-index bytes to every scan — the term this in-repo corpus is too small to price against the comparative gates); the default is not changed here — that is a maintainer decision on the authoritative numbers. The interim OURIOS_COMPACTED_RG_BYTES env knob (RFC 0036 §7) lets an operator retune without a rebuild in the meantime.

9.29 Results — 2026-07-22 (indicative, local M-series, real-corpus subset, Ourios-only, no Loki) — RFC 0036 window-materialization before/after on the v8 capture

Purpose. The real-otel-demo-corpus analogue of §9.27’s synthetic 1.43× before/after — the piece §9.26 structurally could not produce (its single-file-per-partition harness no-ops compact_partition). This measures the RFC 0036 window-materialization win on the actual v8 capture (a subset), Ourios-only. Indicative, not authoritative — local hardware, a corpus subset, no Loki, no baseline VM; the authoritative full-v8 + Loki arm stays deferred (RFC0036.2 §5 / §9.26).

What made this measurable — the opt-in compacted harness path. §9.26’s finding was that ourios-bench’s build_comparative_store writes one ingest file per partition, so compaction no-ops and RFC 0036’s sort never runs. This slice adds an opt-in builder, build_comparative_store_compacted[_with_threshold], that round-robins each partition’s rows across two interleaved ingest files, then calls compact_partition on every partition — so the consolidated file is §3.1-clustered by (service.name, time), rotates row groups, and declares sorting_columns. The default build_comparative_store path — the frozen RFC 0031 dispatch’s store — is byte-for-byte untouched (a separate function; the frozen gates are not re-based).

Hardware / run. Local M-series developer machine (P-cores; taskpolicy -B to escape the Claude-Code background-QoS E-core throttle), dev build. ourios-bench’s rfc0036_realcorpus_window_materialization_before_after (tests/rfc0036_realcorpus.rs, #[ignore]d, skips with a clear message when the gitignored capture is absent so CI/other machines never fail). The store is built two ways from the identical subset — before = default single-file build_comparative_store (no compaction), after = build_comparative_store_compacted_with_threshold — then the same L6-shape window query runs against each.

Corpus / query. First 120,000 LogsData batches of otel-demo-v8/logs.jsonl (513,752,573 B of the 2.96 GB capture, ~21 h of wall-time). The measurement picks the busiest compacted partition (most row groups), then the most-prunable real service in it — which resolves to ad, the lowest-volume otel-demo service and the exact §9.13 run #17 case — and the busy hour as the window: service == "ad" | range(2026-07-07T08:00:00Z, 2026-07-07T09:00:00Z). Materialization bytes = the footer survivor-chunk sum (the RFC 0036 §9 metric: compressed column chunks of the row groups a service = ad ∧ time ∈ window scan cannot prune, keyed on the effective_time_unix_nano column the querier prunes on — not the count-scan stats.bytes_read); the live query’s row_groups_scanned cross-checks the footer prediction.

Reading — a 5.20× real-corpus materialization-bytes win, identical answer. The sort takes the one-service window from materialising the whole hour (the single unsorted ingest row group — nothing prunes) to one of seven clustered row groups (ad lands in a single group; the other six prune on plain footer service.name statistics). Same 10,110-row answer; the live query confirms the footer prediction (before scans 1/1, after scans 1/7). This is larger than §9.27’s synthetic 1.43× — and honestly so: §9.27’s random payload was incompressible and forced the compacted file ~2× larger, whereas real logs compress and the §3.1 service.name sort clusters like lines, so finer clustered groups prune sharply (the §9.28 finding-2 mechanism, now on real data).

The threshold caveat — the then-fixed 32 MiB default did not prune a real hour (§9.28 finding 3 confirmed; fixed by the §9.30 adaptive amendment). Real per-hour v8 volume (~a few MiB compressed) is far below the fixed 32 MiB MAX_COMPACTED_RG_BYTES (the old COMPACTED_ROW_GROUP_FLUSH_BYTES), so at 32 MiB the busy hour compacts to a single row group and nothing prunes — the measurement skips with a “raise subset / lower threshold” message (verified). The win therefore requires a finer compacted threshold (2 MiB here) to rotate the real hour into the several service-clustered groups the pruning mechanism needs. This is not a defect of the layout — it is the same row-cap/volume reality §9.28 finding 3 documented on synthetic data, now reproduced on the real corpus: on realistic per-hour volumes the byte threshold must be finer than 32 MiB to bite. Indicative sensitivity on this subset: 2 MiB → 7 groups, 5.20×; 4 MiB → 2 groups, 1.32×; 32 MiB → 1 group, no prune. The finer the threshold, the sharper the window prune — the §9.28-flagged lever, confirmed on real data.

Disposition (superseded 2026-07-22 — see §9.30). This complements §9.26 (no-regression, but structurally could not show the win) and §9.27 (synthetic 1.43×) with a real-corpus before/after (5.20× at an explicit 2 MiB threshold). Its central finding — that the fixed 32 MiB shipped default is inert on a real per-hour v8 hour (compacts to one group, prunes nothing, the measurement skipped), and the win only appears at a hand-set finer threshold — is exactly what motivated the RFC 0036 §3.3 adaptive-threshold amendment (2026-07-22): the threshold now scales as clamp(input_total / 8, 1 MiB, 32 MiB), so a small real hour floors at 1 MiB and prunes without any operator tuning. §9.30 re-runs this same measurement at the adaptive default and records the win there. The authoritative full-v8 L6-scanned-bytes-vs-Loki arm, and the ceiling/target-K sweep, stay deferred to the paid baseline-8vcpu-32gib harness (RFC 0036 §7 / RFC0036.2).

9.30 Results — 2026-07-22 (indicative, local M-series, real-corpus subset, Ourios-only, no Loki) — RFC 0036 window-materialization at the ADAPTIVE default

Purpose. §9.28/§9.29 established that a fixed compacted row-group threshold is the wrong shape: 32 MiB is inert on a real per-hour v8 hour (a few MiB compressed → one row group → nothing prunes; §9.29 skipped at the 32 MiB default), while the same fixed value fragments large hours. The RFC 0036 §3.3 adaptive amendment (2026-07-22, maintainer-approved) replaces it with adaptive_flush_bytes = clamp(estimated_output_bytes / 8, 1 MiB, 32 MiB) — target ~8 groups per partition, floored at 1 MiB (the lever for small hours) and capped at 32 MiB (the old fixed value, now the ceiling for huge hours). This run is the decisive check: re-run §9.29’s real-corpus before/after at the adaptive default (OURIOS_V8_COMPACTED_RG_BYTES unset) and confirm the win now appears at the shipped default — not only under a hand-set threshold. Indicative (local, subset, Ourios-only, no Loki), same caveats as §9.29.

Hardware / run. Local M-series developer machine (P-cores; taskpolicy -B), dev build. ourios-bench’s rfc0036_realcorpus_window_materialization_before_after (tests/rfc0036_realcorpus.rs, #[ignore]d), threshold env unset so the adaptive default governs. Same corpus and same selection procedure as §9.29 (not a fixed query — the harness derives the target/window from the data): the first 120,000 LogsData batches (513,752,573 B) of otel-demo-v8/logs.jsonl, then it dynamically picks the busiest compacted partition (most row groups), the most-prunable real service in it — which again resolves to ad (lowest-volume otel-demo service, the §9.13 run #17 case) — and that partition’s own time span as the window. Here the selected partition is the 07:00–08:00 hour, so the query is service == "ad" | range(2026-07-07T07:00:00Z, 2026-07-07T08:00:00Z) (a different hour from §9.29’s dynamically-selected 08:00–09:00 — expected, since the partition is re-chosen from the store the adaptive build produced, not hard-coded). Materialization bytes = the footer survivor-chunk sum (RFC 0036 §9 metric, keyed on effective_time_unix_nano); the live query’s row_groups_scanned cross-checks the footer prediction.

Reading — a 7.61× real-corpus materialization-bytes win at the shipped default, identical answer. With the threshold unset, the adaptive value floors at 1 MiB for this small real hour and rotates it into 21 service-clustered row groups; the ad window lands in a single group and the other 20 prune on plain footer service.name/time statistics. The sort takes the one-service window from materialising the whole hour (3,812,581 B, one unsorted group) to one clustered group (500,537 B) for the identical 9,984-row answer — the live query confirms the footer prediction (before scans 1/1, after scans 1/21). This is the point of the amendment: the win now appears at the shipped default, where the fixed 32 MiB threshold had skipped (§9.29). It is even sharper than §9.29’s explicit-2 MiB 5.20× — the 1 MiB adaptive floor is finer still, so it clusters into more, tighter groups.

Disposition. The adaptive default makes RFC 0036 non-inert on real v8 — the fix §9.28/§9.29 pointed at, now measured. The rfc0036_2_* in-repo gates recompute T = adaptive_flush_bytes(input_total) and track the layout whatever it resolves to. The authoritative full-v8 L6-scanned-bytes-vs-Loki arm and the ceiling/target-K sweep stay deferred to the paid baseline-8vcpu-32gib harness (RFC 0036 §7 / RFC0036.2).

Roadmap to MVP

Living document. Refreshed at phase boundaries (§4) and whenever a merged PR materially changes the current state in §3. Last updated: 2026-07-27 — the typed-promotion arc landed end-to-end and the dashboard decision is made. RFC 0042 (typed numeric promotion, RFC 0022 §7.1 enacted) went spec → green in two days (#647–#653), closing with RFC0042.9 verified on live telemetry: the agent queried its own spend over MCP — the sum(attr.cost_usd) by attr.model query returned 35.28 USD across 10 requests from the typed Float64 column. Getting the capture to flow fixed a latent env bug (#654: per-signal OTLP endpoints are used as-is per spec, so the dogfood env now carries explicit /v1/<signal> paths). RFC 0041 went specified → green the same date: build now, Perses first — the three plugins shipped in the dedicated ourios-perses-plugin repository (PRs #1–#6), and the capstone FinOps dashboard (RFC0041.6, examples/perses/, #661) rendered unmodified against the live dogfood capture. One recorded deferral: RFC0041.5’s latest matrix leg waits for the next server release (the first with typed columns); Grafana remains an ungated follow-up. The unreleased breaking changes on main since v0.5.0: #641 (RFC0002.21 severity) and #645 (Helm otel values) — the next tag is not a patch.

Prior entry — 2026-07-25: the self-observability arc landed and the RFC 0036 arc closed. RFC 0038 (self-tracing), RFC 0039 (inbound trace-context propagation) and RFC 0040 (DataFusion operator instrumentation) are all green: Ourios continues a caller’s trace instead of starting its own, and a finished physical plan is reconstructed post-hoc into a span tree by ourios-df-otel — a crate carrying only datafusion + opentelemetry, kept extractable for upstream. RFC 0037 (GenAI / structured-event logs) is green. v0.5.0 shipped. §3’s ladder now covers RFC 0001 through RFC 0041. RFC 0041 (dashboard datasource plugins) was drafted with §5/§6 deliberately empty — its §7 asked whether the work is worth doing at all. (Resolved in the current entry: yes, Perses first.)

One unreleased breaking change sat on main behind v0.5.0 at this entry: RFC0002.21 (#641), unspecified severity aligned with the OTel SDK. (The current entry tracks the full unreleased-breaking list.)

Prior entry — 2026-07-21: the comparative program closed and the ingest-capacity arc landed; §3’s ladder now covers RFC 0001 through RFC 0036. RFC 0031’s first fully authoritative comparative run (baseline-8vcpu-32gib, benchmarks.md §9.24) passed all 11 frozen gate decisions — L1 97.82× / L3 22.52× storage-primary, L2 38.37× / L4 85.14× processed-primary, both L6 latency floors — and the RFC flipped to validated (accepted is a maintainer flip). The D1 arc shipped alongside: RFC 0034 (specified, enacted) recast D1 as a per-node bar, and RFC 0035 (green) split ingest into an ordered mining phase and a concurrent encode/publish phase to clear it — the §9.23 asserting soak holds an offered 100k lines/s per node (99.92% achieved, p99 153.63 ms) on the baseline hardware. RFC 0036 (specified) opens the next arc: write-side layout (compaction-time service/time sort), the remaining storage lever against hazard #4. (That arc has since closed — see the current entry.)

Prior entry — 2026-07-15: a month of post-MVP shipping work landed since the prior entry below; §3’s RFC ladder now covers RFC 0001 through RFC 0033 and §5’s deferred-capabilities table (eight rows) is rewritten: six have shipped outright (the WAL, the OTLP wire endpoints, the snapshot mechanism, the §6.8 telemetry surface, the query DSL, and the ourios-server binary + Helm chart), multi-tenancy-at-runtime is partially landed (auth + tenant binding shipped via RFC 0026 accepted; rate-limit/ eviction/lifecycle orchestration is still open), and the Perses datasource plugin remains fully deferred. Current work is RFC 0031 (comparative evaluation against Grafana Loki) — a post-MVP thesis-strengthening effort, not a new MVP gate — with the L1/L3/L6 classes frozen and gate-enforcing per its §7 and the last must-win class (L4, frequency aggregation) mid-dispatch. Phases 1–3 in §4 are all complete; that section is historical narrative only from this point forward.

Prior entry — 2026-06-15: RFC 0013 (object storage, S3-compatible) drafted → specified → red (first shipping-milestone spine; store module skeleton + §5 stubs landed); RFC 0009 (background compaction) flipped to validated (RFC0009.7 D2/D3/B2-post measured on baseline-8vcpu-32gib, §9.7); RFC 0005 (Parquet storage) and RFC 0010 (audit-stream / drift queries) flipped to green (RFC0005.6 row-group sizing landed; RFC 0010’s eight §5 drift scenarios all pass). Earlier, on 2026-06-14, RFC 0001, RFC 0008, and RFC 0011 flipped to accepted (maintainer sign-off). RFC 0001 reached validated first (C1/C2 pass authoritatively on the benchmarks.md §1 baseline hardware, §9.6; A1 is diagnostic per RFC 0011); RFC 0008’s validated is vacuous (no thesis gate); RFC 0011 is a tuning RFC. The §§4+ phase narrative below predates this and is not re-verified here (PR #41 RFC 0005, then PR-D through PR-G landed ourios-parquet end-to-end: schemas, writer, reader, audit stream). The deferred-capabilities table in §5 is unchanged: WAL durability and the OTLP wire endpoints stay post-MVP.

This document answers two questions in one place: what does “MVP” mean for Ourios, and how far are we from it. The artifact is parallel to hazards.md and benchmarks.md: hazards say what we mustn’t break, benchmarks say what success looks like, and this file says how we get from here to there.

1. What “MVP” means here

MVP for Ourios is thesis-proving, not production-ready.

The thesis (CLAUDE.md §2) claims that Parquet + Drain-derived template mining + DataFusion collapses the inverted index, the compression layer, the storage tier, and the query engine into one stack of off-the-shelf parts plus thin glue. That claim is falsifiable. The MVP is the smallest stack that lets us run the thesis-gate benchmarks in benchmarks.md on a real corpus and either confirm the claim or kill it.

Production-shape concerns — gRPC OTLP receiver, WAL durability, snapshot mechanism, Helm chart, the full §6.8 telemetry surface, the RFC 0002 query DSL — are deliberately out of MVP scope (§5). Each is a real shipping concern, but none of them changes the answer to “does the thesis hold.” We defer to keep the critical path as short and honest as possible.

2. The MVP gate: thesis benchmarks

Four gating [THESIS] goals in benchmarks.md define MVP-done. Hitting all four on a representative corpus means the thesis holds; missing any of them means a pillar (CLAUDE.md §2) is wrong and a PR won’t fix it — an RFC will.

A1 (end-to-end compression vs. zstd-alone) was a fifth gating goal, but RFC 0011 (accepted) demoted it to a recorded diagnostic: it is refuted on every corpus class — including the maximally-templated one — for structural reasons (the more templated a corpus, the more a whole-stream byte codec captures the same redundancy), so template mining’s compression value is logical / query-pruning, captured by B1/B2, not on-disk bytes vs a codec. A1 is still measured and recorded (benchmarks.md §7/§9 — the columnar queryability premium + a codec-regression guard) but does not block MVP-done or any RFC’s validated.

A2, B3, C3, C4, D*, E* in benchmarks.md are relevant but not MVP-blocking — they’re tuning goals, honesty goals, or post-MVP shipping concerns.

3. Current state (as of 2026-07-25)

The thesis is proven on representative corpora. All four gating thesis-gates pass authoritatively on the benchmarks.md §1 baseline hardware (the §9.4 / §9.6 runs), so the MVP thesis-proving bar (§2) is met:

A1 (compression vs zstd) fails, but RFC 0011 (accepted) reclassified it a recorded diagnostic, not a gate: the failure is structural and template mining’s value is logical / query-pruning, captured by B1/B2 (see benchmarks.md §2 / §7).

RFC ladder status:

Crates — all twelve product crates are implemented (ourios-core, -config, -miner, -wal, -parquet, -ingester, -querier, -server, -bench, -semconv, -telemetry, -df-otel; a thirteenth, -testgen, is dev-only):

• ourios-miner — the Drain-derived miner, RFC 0001 accepted: (severity, scope) keying, three-zone confidence, widening + type-expansion with audit events, 256 B param-overflow spill, bit-identical reconstruction + the H7.3 render contract, structured-body canonical encoding, and §6.9 snapshot + v2 restore. Zero #[ignore]/todo!() acceptance stubs.
• ourios-wal — RFC 0008 accepted: append/sync, crash recovery (the real-SIGKILL CI gate), snapshot-restore, segment rotation, group-commit batched fsync, checkpoint-driven truncation; §5 arms .1–.10 green.
• ourios-parquet — RFC 0005 §3: atomic-publish writer + reader with the §3.9 compat contract, the §3.7 audit-event series, and the §3.6 encoding policy (dict + page index + template_id bloom filter).
• ourios-ingester — RFC 0003 green: the OTLP gRPC + HTTP receiver with WAL-before-ack, per-ResourceLogs tenant derivation, the windowed group-commit coordinator, and the startup recovery driver; also hosts the RFC 0009 compaction runner and the RFC 0035 two-phase pipeline (ordered mining, concurrent encode/publish).
• ourios-querier — RFC 0007 validated / RFC 0002 green: the logs DSL over DataFusion with predicate + partition (time-window) pruning, alias resolution, the RFC 0010 drift query, param(n)/bucket(width) aggregation (RFC 0002’s L4 amendment), and the RFC 0032 query-schema + cost-model MCP resource.
• ourios-bench — RFC 0006 green: drives the A1/B1/B2/C1/C2 measurements over OTLP-Demo + LogHub corpora, records results to benchmarks.md §9, and (RFC 0031) runs the comparative dispatch against a real Loki container.
• ourios-df-otel — RFC 0040 green: a post-hoc ExecutionPlan → OTel span-tree walk, backdating each operator span from its BaselineMetrics timestamps. Its runtime dependencies are datafusion and opentelemetry alone — no ourios-* crate among them — so it stays extractable as an upstream contribution.
• ourios-core / -config / -semconv / -telemetry / -server — shared types + tenancy + record/audit shapes; the RFC 0004 miner tunables (split out per RFC 0028 §3.2); the weaver-generated OTel name constants; the OTel export surface — metrics (RFC 0018) and, since RFC 0038/0039, traces; the two-role binary, now with TLS/mTLS (RFC 0030), an OIDC bearer layer (RFC 0029), and the S3-native Helm chart, deploy-validated on kind.

The full cargo test --all-features suite is green in CI — the cargo test job gates every PR on the exact head; the coverage job runs alongside it but is informational (continue-on-error), not gating.

What remains is no longer “reach MVP” — that bar cleared a month before this entry (§3’s gate table is unchanged and still authoritative) and the shipping milestone that followed (WAL, wire endpoints, DSL, auth, S3, Helm — the whole §5 table below except Perses) is substantially done. What’s actually open:

• The Perses plugins — RFC 0041 is accepted (2026-07-28, terminal): all three plugins shipped in ourios-perses-plugin, the RFC0041.6 dashboard is committed in this repo and verified rendering. What remains is RFC0041.5’s recorded deferral (the latest e2e leg + wire-level sum, unblocked by the next server release, tracked in the plugin repo) and the Grafana datasource as an ungated follow-up.
• RFC 0040 → accepted — done (2026-07-28), alongside RFC 0041 and RFC 0042: all three flipped accepted on maintainer sign-off.
• Scattered §7/§9 open items on already-green/validated RFCs (e.g. the recurring D1/D2 soak cadence now that the harness has shipped (§9.19/§9.23), RFC 0021’s phase 2 gated on upstream DataFusion 55, RFC 0028’s musl cargo-dist re-add, RFC 0031’s deferred F_L7) — none block anything downstream.

4. Path to MVP — three phases

Phase scope only; per-PR breakdown lives in the planning that opens each phase, not in this doc, so the file stays stable as mid-stream design decisions land.

Phase 1 — Finish the miner

Goal: the miner mines, audits, retains bodies, reconstructs. By the end of this phase the miner self-contained covers RFC 0001 §6.2 / §6.3 / §6.4 / §6.5 / §6.6 end-to-end and most §5 scenarios are green.

Capabilities to land:

• Drain tree (root → length-N nodes → prefix nodes → leaves) with descend.
• Best-candidate selection in MinerCluster::ingest via sim_seq (replaces the exact-match HashMap placeholder).
• widen step + template_widened audit emission + type-expansion + template_type_expanded audit + degenerate- template guard.
• Three-zone confidence branching (clean / lossy / parse-failure)
  • body retention in the lossy zone.
• Separators preservation through the ingest pipeline + reconstruct() + lossy_flag semantics per §6.6.
• Per-parameter byte-limit check + OVERFLOW marker + forced body retention.
• MinerCluster::ingest consumes a structured OtlpLogRecord (per RFC 0001 §6.1 as amended), not a raw &str. The body_kind = String / body_kind = Structured fork lands with the §6.2 algorithm rewrite (a follow-on PR to the §6.1 amendment). Severity, scope, and the OTLP-canonical JSON encoding for structured bodies all flow through the miner from this phase forward.

Unblocks: thesis gates C1 (reconstruction) and C2 (template-count convergence). RFC 0001 §5 scenarios H1.*, H2.*, H5.*, H7.*, §3.3.1, RFC0001.* should mostly flip in this phase.

Phase 2 — Records to Parquet

Goal: mined records become Parquet files. By the end of this phase a corpus run produces on-disk Parquet that any DataFusion-aware reader can open.

Capabilities to land:

• New crate ourios-parquet.
• Record schema matching the amended RFC 0001 §6.1: identity + partitioning columns, the OTLP-derived columns (time_unix_nano, severity_number + severity_text, scope_name + scope_version, attributes, resource_attributes, trace_id + span_id + flags, event_name, dropped_attributes_count), and the body / miner-derived columns (body_kind, body?, params, separators, confidence, lossy_flag).
• Writer: record batch → Parquet file (with row-group sizing from hazards.md H4 — target 128 MB–1 GB row groups).
• Reader: Parquet file → record batch (for verification + the Phase 3 DataFusion path).
• Audit-event Parquet stream (the contract called out in RFC 0001 §9 “Cross-RFC contracts pending”).

Unblocks: thesis gate A1 (compression ratio). The Parquet column codec earns its share of the 50–200× headline only once records actually land on disk in this format.

Out of MVP scope, parked here: background compaction (small-file problem, hazards.md H4) — corpus runs are bounded, a single Parquet file per phase is acceptable; production compaction is a post-MVP PR.

Phase 3 — DataFusion + bench

Goal: the thesis-gate benchmarks run.

Capabilities to land:

• New crate ourios-querier — register the Phase 2 Parquet files with DataFusion and accept raw SQL. No DSL — RFC 0002’s surface is a post-MVP concern; the bench can use SQL directly.
• New crate ourios-bench — corpus runner that reads pre-recorded OTLP LogsData test data into a stream of OtlpLogRecords, hands them to the miner, writes Parquet, runs the A1/B1/B2/C1/C2 measurements, and reports numbers that go into benchmarks.md §9 (Status). No network receiver in MVP — the bench reads OTLP from disk, not from a gRPC/HTTP listener (those stay post-MVP per §5).
• testdata/corpus/ — anonymised real-log corpus committed to the repo (or a download script if size demands), serialised as OTLP LogsData (canonical JSON or protobuf) so the bench exercises the same record shape an OTel deployment would produce.

Unblocks: thesis gates B1 (predicate-pushdown latency) and B2 (template-exact latency). At the end of this phase, benchmarks.md §7 (the thesis-gate summary) has measured numbers for every [THESIS] row, and either the thesis holds or it doesn’t.

5. Deliberately out of MVP

Each item is a real production concern. The reason it was deferred for MVP is “answering ‘does the thesis hold?’ doesn’t require it,” not “we don’t think it matters.” As of this entry, seven of the eight original rows have shipped outright — the Perses row’s plugin half landed with RFC 0041 green (CRDs/operator stay ungated) — and one (multi-tenancy at runtime) is partially landed, as part of the post-MVP shipping milestone (§3); the table below records what shipped and what’s still genuinely open.

Note on OTLP scope (historical). The pre-amendment roadmap listed “OTLP receiver (gRPC + HTTP)” as a single post-MVP item. PR #20 + #21 split that scope: the OTLP record shape (OtlpLogRecord consumption, the canonical JSON encoding, the OTLP-aligned Parquet schema) was in MVP from the start — it’s a prerequisite for thesis-gate C2’s validity, because the template-count convergence the corpus measures has to be over records that look like real OTel traffic, not over flat-text caricatures of it. Only the wire endpoints — the actual gRPC/HTTP listeners that decode OTLP off the network — were deferred past MVP, and that’s the row in the table above; RFC 0003 (green) has since landed them, so nothing in this note is still-open scope.

6. Update cadence

This file refreshes:

• After every merged PR that materially changes §3 (current state) — the merging PR’s author (or their drafting assistant) updates the table and the §5 scenario count.
• At phase boundaries (§4) — when Phase 1 finishes, §3’s current state and §4’s “blockers” tables are reconciled, and the next-phase opening planning PR is summarised here.
• When a thesis-gate result lands in benchmarks.md §9 — this doc gets a one-line note in §3 acknowledging the result.

The doc is intentionally not refreshed on every spec edit — RFC patches and hazards.md edits don’t change the road map unless they change what MVP requires. If you find yourself updating §3 every PR, the doc has become an activity log; the fix is to be more selective, not to stop updating.

RFCs

Referenced from CLAUDE.md §5.1. This document is the minimum viable RFC process for Ourios. It will grow as the project does.

When an RFC is required

Per CLAUDE.md §5.1, an RFC precedes implementation for any change that touches:

• An architectural pillar (CLAUDE.md §2).
• An invariant (CLAUDE.md §3).
• A hazard (CLAUDE.md §4 / docs/hazards.md).
• The on-disk Parquet schema (CLAUDE.md §3.5).
• A new crate (CLAUDE.md §7).

Bug fixes, dependency bumps, and internal refactors do not need RFCs. When in doubt, assume RFC.

File layout

• Filename: NNNN-short-kebab-title.md, e.g. 0001-template-miner.md.
• Numbers are assigned in merge order. Draft PRs may use the next free number provisionally; if two drafts collide, the later-merged one renumbers.
• One file per RFC. Supersessions are recorded in the frontmatter of both the old and new RFC.

Required frontmatter

---
rfc: NNNN
title: Short descriptive title
status: drafted | specified | red | green | validated | accepted | rejected | superseded
author: Name <email>
drafting-assistance: Claude   # omit if no LLM drafted
created: YYYY-MM-DD
supersedes: —                 # or RFC NNNN
superseded-by: —              # or RFC NNNN
---

The maturity stages (drafted through validated) are gates an RFC moves through before it becomes binding; accepted is the terminal post-maintainer-signoff state; rejected and superseded are the off-ramps. See docs/verification.md §3.

Required sections

Every RFC has at least:

1. Summary — 3–5 sentences. The commitment, not the rationale.
2. Motivation — why this change now, and why at this layer.
3. Proposed design — precise enough that two engineers would produce the same implementation.
4. Alternatives considered — one paragraph each. “I have not heard of it” is not acceptable.
5. Acceptance criteria — normative scenarios, one per invariant or hazard the RFC touches. Format: structured prose with Given / When / Then / And leading clauses; each scenario carries an id of the form H1.1, §3.4.2, or RFC<NNNN>.<m>, referenced from the test code so the mapping is greppable. See docs/verification.md §2.
6. Testing strategy — mapped to CLAUDE.md §6.2; references the §5 scenario ids and names the technique (proptest, corpus, criterion) for each.
7. Open questions — everything unresolved, as a checklist.
8. References — paper citations, related RFCs, CLAUDE.md sections constrained.

Additional sections are welcome when they clarify. Do not pad for the sake of the template.

Lifecycle

The five-stage maturity model. An RFC moves through these stages before becoming binding; the status: frontmatter field tracks the current stage so reviewers and tooling see it without reading the body.

1. Drafted — PR opened with status drafted. Sections §§1–4 and §§7–8 are filled. Discussion happens in PR review.
2. Specified — §5 acceptance criteria are written, every invariant and hazard the RFC touches has at least one scenario, and review has confirmed the criteria are testable in principle.
3. Red — test stubs exist and fail. Implementation may begin.
4. Green — all acceptance criteria pass; unit + property + corpus tests green.
5. Validated — thesis-gates in docs/benchmarks.md §7 pass on representative corpora. Maintainer flips status to accepted.

A regression detected after Validated either reopens the RFC (if a criterion is invalidated) or spawns a tuning RFC per benchmarks.md §7 (if a thesis-gate degrades). See docs/verification.md §3.

Two terminals reachable from any stage:

• Superseded — a later RFC replaces part or all of this one. Both frontmatters are updated. The superseded RFC is not deleted.
• Rejected — closed PR or status flipped to rejected. The file is kept for the record.

Diagrams

When an RFC needs a diagram (state machine, sequence flow, schema relationship, decision tree), it is authored in Mermaid, embedded as a fenced ```mermaid block in the markdown. Mermaid is chosen for the same reasons we chose markdown over a binary doc format: text-based source is reviewable in PR diffs, version-controllable, and lets the RFC itself remain a single self-contained file.

Lectures (docs/talks/) use a different convention: hand-drawn SVGs (Excalidraw export, or hand-authored to match) committed under docs/talks/img/. Lectures benefit from a “manuscript / blackboard” aesthetic that Mermaid does not provide; RFCs benefit from the diff-ability that Excalidraw does not provide. Do not mix the two conventions.

The mdBook build has the mdbook-mermaid preprocessor enabled (book.toml), with the Mermaid runtime vendored at the repo root (mermaid.min.js, mermaid-init.js) so the rendered book is self-contained. The CI book job and the Pages workflow install the mdbook-mermaid binary before building. To work on diagrams locally, cargo install mdbook-mermaid --locked (the preprocessor binary) — the vendored runtime is already committed.

Relationship to architecture docs

An accepted RFC is a contract for how something will be built. Once the subsystem is stable, the RFC graduates to docs/architecture/<subsystem>.md — a living document describing the system as it actually is. The RFC stays in place as the historical decision record; the architecture doc is what a new contributor reads first.

RFC 0001 — Template miner

rfc: 0001 title: Template miner (Drain-derived online log parsing) status: accepted author: Jens Holdgaard Pedersen jens@holdgaard.org drafting-assistance: Claude created: 2026-04-24 supersedes: — superseded-by: —

RFC 0001 — Template miner

Status note. accepted (2026-06-14, maintainer sign-off — the terminal ladder status per docs/rfcs/README.md). Reached validated the same day on the evidence below; accepted records the maintainer’s final sign-off on the template-mining pillar. The docs/verification.md §3 / docs/rfcs/README.md ladder reserves validated for every thesis-gate the RFC’s pillars touch passing on representative corpora (benchmarks.md §7). The template-mining pillar’s gates are C1 (reconstruction fidelity) and C2 (template-count convergence) — both pass on the representative LogHub HDFS_v1 corpus (≈ 1.47 GiB, 11.2 M lines — above benchmarks.md §8’s ≥ 1 GiB canonical floor, so representative; and well past C2’s ≥ 1 M-line formal-gate threshold, so that gate applies rather than abstains), authoritatively on the benchmarks.md §1 baseline hardware: C1 1.000000, C2 a 40-template plateau (diagnostic local run benchmarks.md §9.5, authoritative baseline-8vcpu-32gib rerun §9.6 — identical verdicts, as expected of deterministic gates). A1 (compression vs zstd) is a diagnostic, not a gate (RFC 0011, already encoded in the §7 gate table): it fails on every corpus class including the maximally-templated one, for structural reasons — template mining’s compression is logical/query-pruning (B1/B2), not on-disk bytes.

Reached green first (2026-06-13): all §5 acceptance criteria pass with live tests (zero #[ignore]/todo!() stubs) — miner-internal (tokenize/mask/sim-seq, the three-zone confidence model, fresh-leaf + widening + type-expansion with audit events, the (severity_number, scope_name) template key, the 256 B param-overflow spill + telemetry, bit-identical reconstruction + the H7.3 render contract, structured-body canonical encoding, the §6.9 snapshot + v2 restore) plus the relocated cross-crate criteria (query semantics RFC0001.5/.6, time-preserved RFC0001.10, §3.7.3 per-ResourceLogs tenant derivation, drift H5.3 via RFC 0010, the §6.7 alias index via RFC 0005 §3.7).

Terminal step: accepted is the maintainer’s final sign-off (docs/rfcs/README.md). NB the A1 re-scope’s own RFC (RFC 0011) is still drafted; the §7 gate table already reflects the demotion, but RFC 0011 should be accepted to fully ratify that chain.

How to read this document. §§1–4 are the design contract — the what and the why. §5 lists the normative Given / When / Then scenarios — the contract — grouped by parent (hazard, invariant, RFC-internal). §6 is the precise specification the ourios-miner crate is implemented against; its opening paragraphs name the gaps between the published algorithm and a production miner that §6.1–§6.9 then close. §7 records the alternatives we evaluated and rejected. §8 maps each §5 scenario to the technique that tests it.

Cross-references to CLAUDE.md sections are in square brackets, e.g. [§3.1], and name the invariant the section must preserve.

1. Summary

Ourios implements a Drain-derived online template miner (ourios-miner) that converts each ingested OTLP LogRecord into a structured Parquet record. The record shape is the OTLP LogRecord (with its inherited Resource and InstrumentationScope context) plus the miner-derived columns (template_id, template_version, params, separators, body_kind, body?, confidence, lossy_flag); see §6.1 for the full schema and the 2026-05-13 amendment that aligned it to OTLP. The miner is per-tenant by construction [§3.7], uses a three-zone confidence model that retains the original line in the lossy zone [§3.1], audits every template widening [§3.1], captures inter-token separators in a parallel array so that bit-identical reconstruction is the default rather than a property-test exception [§3.3], bounds parameter values at 256 B with overflow to a side body column [§3.2], and tracks template structural changes via a monotonic template_version so that schema drift across deploys is a first-class query rather than a silent count drop [§3.5]. The 50–200× figure is a logical reduction (lines → (template_id, params)), realised as query pruning (gates B1/B2), not on-disk bytes versus a byte codec — see RFC 0011, which demoted the compression-vs-zstd ratio (A1) to a diagnostic.

2. Motivation

This is the load-bearing pillar of the project [§2.2]. Three sub-questions justify it.

Why template mining at all. A typical service emits 10²–10⁴ distinct printf templates over its entire lifetime, but raw log volume is dominated by the parameters substituted into those templates. Storing the template once per tenant and the parameters per occurrence makes that redundancy explicit — and explicit redundancy stacks with byte-level codecs rather than fighting them. zstd over flat log text recovers ~10× on typical workloads; doing the structural work first leaves zstd a column of short, repetitive parameters that dictionary-encode well, where the codec then earns its keep again. The 50–200× headline (README.md, [§2.2]) is the product of these two layers, not a claim about either alone.

Why online vs. offline. Operators expect logs to be queryable within seconds of ingest, not minutes. Any batch clustering window long enough to do offline hierarchical clustering well is a window the operator is blind in. Drain’s fixed-depth tree gives O(d) lookup per line at the cost of being slightly less accurate than the best offline parsers — an acceptable trade because §3.1’s audit and confidence machinery surfaces the inaccuracy rather than hiding it.

Why this layer. The compression is structural, not statistical. Doing it before Parquet’s byte codecs means each Parquet column sees small, dictionary-friendly values; doing it after means we have already paid for storing the redundancy and zstd has to find it again from the bytes. The order matters.

3. Background: Drain as published

A restatement of He et al., ICWS 2017, in the notation this RFC uses downstream. Citations are by paper section/figure.

3.1 Tree structure (paper §3.2, Fig. 2)

A fixed-depth parse tree, depth d (default 4 in the paper). Three node kinds, in order from root:

• Root. Single node; routes by token count.
• Length-N node. One per observed token count N. Children are prefix nodes keyed by the first token of the line.
• Token-prefix nodes at depths 2..=d. Each is keyed by the token at position (depth - 1) of the line.
• Leaf log groups at depth d + 1. Each leaf holds a template — a sequence of N tokens, where each position is either a fixed string or the wildcard <*>.

                   root
                    │
        ┌───────────┼───────────┐
       len=4       len=5       len=6      ← length groups
        │           │           │
    ┌───┴───┐    ┌──┴──┐     ┌──┴──┐      ← prefix nodes (depth 1)
   "user"   …  "GET"  …    "INFO"  …
    │           │           │
   ┌┴┐         ┌┴┐         ┌┴┐
   …  …        …  …        …  …          ← prefix nodes (depth 2)
   │           │           │
 [leaf]      [leaf]      [leaf]           ← log groups

3.2 Similarity function (paper §3.3)

For a candidate line L = (t_1, …, t_N) and a leaf template T = (τ_1, …, τ_N):

simSeq(L, T) = (count of positions i where t_i == τ_i or τ_i is <*>) / N

Wildcards in the template count as matches. The line length and the template length are equal by construction (the length-N node selected the leaf candidates).

3.3 Threshold st (paper §3.4)

A configured value st ∈ (0, 1]. After computing simSeq against every leaf at the current parent, the leaf with the highest simSeq is the candidate. If simSeq(L, T_best) ≥ st, the line attaches to T_best; otherwise a new leaf is created. The paper reports st = 0.4 as a default; see §6.3 for why Ourios overrides this.

3.4 New-log-group creation vs. leaf update (paper §3.5)

If simSeq(L, T_best) < st, a new leaf is created at the parent prefix node, with L as its initial template (no wildcards yet). Otherwise T_best is updated: at every position where t_i ≠ τ_i, the template position is replaced with <*>. The template never becomes more specific over time, only more general; positions can become wildcards but cannot become fixed again.

3.5 Worked example

A fabricated illustration (no testdata/corpus/ exists yet; this example will be replaced with one drawn from the corpus once it lands).

Line A:  user 42 logged in from 10.0.0.1
Line B:  user 17 logged in from 10.0.0.2
Line C:  user 99 logged out from 10.0.0.7

After preprocessing (§4.2), numbers and IPs are masked, so the miner sees:

Line A:  user <NUM> logged in from <IP>
Line B:  user <NUM> logged in from <IP>
Line C:  user <NUM> logged out from <IP>

All three are length 6. They route to root → len=6 → "user".

A walks the prefix path further (depth 2: token at position 1 is the masked <NUM> placeholder, treated as a fixed token at this stage). It is the first line, so a leaf is created with template user <NUM> logged in from <IP>.

B walks the same path. The candidate leaf has simSeq(B, T_A) = 6/6 = 1.0 ≥ st. B attaches; the template is unchanged.

C walks the same path. The candidate leaf has simSeq(C, T_A) = 5/6 ≈ 0.833. With st = 0.7 (Ourios default, §6.3), 0.833 ≥ 0.7, so C attaches. Token position 4 (in vs out, 1-indexed against the masked sequence) becomes <*>. The template widens to user <NUM> logged <*> from <IP>. This is a template widening event and must emit an audit record per §6.4.

4. Background: Drain3 extensions (not in the paper)

Drain3 (logpai/Drain3) is the maintained Python implementation. It adds several capabilities beyond the 2017 paper. Each is recorded here as adopt, adopt with modification, or reject, with one sentence of rationale.

4.1 Persistent state — adopt with modification

Drain3 supports JSON snapshots to file, Redis, or Kafka. Ourios adopts the snapshot concept but commits to a file/object-storage backend; Redis and Kafka are out of scope (CLAUDE.md §3.6 names object storage as the source of truth). Snapshot target, cadence, and scope are open questions in §9.

4.2 Pre-tree-walk masking — adopt with modification

Drain3’s most important extension: regex-based masking of common parameter shapes (IPs, UUIDs, numbers, hex, timestamps, file paths) before the tree walk, so high-cardinality tokens never become tree branches. Without this, the tree explodes into one branch per IP address.

The Ourios modification: a masked token is not discarded. It becomes a typed parameter attached to the wildcard slot it created. The masking layer emits (type_tag, original_bytes) pairs; the tree walk treats the type tag as the token (so <NUM> matches <NUM> for tree-routing purposes) while the original_bytes flow into params so reconstruction can recover the line exactly. Paper-pure Drain loses the original token; Ourios retains it as a parameter. This is what makes [§3.3] reconstruction possible at all.

4.3 Variable-length wildcards — adopt with constraint

Drain3’s MaskingInstruction allows a single regex to match a variable-length run of tokens (e.g. a multi-token user-agent string). Ourios adopts this where the run is bounded at parse time and produces exactly one typed parameter in the output. Reject: unbounded variable-length wildcards, because they break leaf identity (two lines with the same template structure but different run lengths would land in different length-N nodes and never deduplicate).

4.4 Dynamic / adaptive threshold — reject

Drain3 supports auto-tuning the similarity threshold per leaf based on observed cluster sizes. Ourios rejects this. CLAUDE.md §3.1 fixes threshold ≥ 0.7 as a project-level invariant; auto-tuning would silently move the merge boundary across deploys, defeating the audit contract. Threshold tuning is a config decision per tenant, never a runtime decision per leaf.

4.5 Other Drain3 features

• Parameter-naming hints. Drain3 lets users name <*> slots via the masking config (e.g. <IP:client_addr>). adopt — the type-tag mechanism in §4.2 already requires a slot name; using the Drain3 hint format keeps configs portable.
• Built-in metrics surface. Drain3 exposes a set of state counters via callback. replace — Ourios exposes OTel metrics directly per §6.8 (instrumented via the meter API), with names that match [§3.1]’s required set rather than Drain3’s internal names.
• Parameter masking after the fact. Drain3 has utilities to retroactively mask params in already-clustered lines. reject — Ourios masks once, at ingest, deterministically. Retroactive masking would invalidate already-written Parquet files.

5. Acceptance criteria

Per docs/verification.md §§2–3, every CLAUDE.md §3 invariant and every docs/hazards.md hazard this RFC touches has at least one numbered scenario below. Scenarios use the bold-leading-clause format (verification.md §2.1) and the id grammars (§2.2): H<n>.<m> for hazard-rooted, §3.<n>.<m> for invariant-rooted, RFC0001.<m> for design-internal commitments. Test code carries each id in a doc comment per §2.3 so grep -R "H1.1" . resolves bidirectionally between RFC and tests.

The hazards in scope are H1, H2, H5, H7; the invariants are §3.1, §3.2, §3.3, §3.5, §3.7. H3 (WAL durability) and H4 (small files) are owned by the ourios-wal and ourios-parquet RFCs; H6 (DSL) is owned by RFC 0002; §3.4 (WAL-before-ack) and §3.6 (object-storage-as-truth) are touched only via §6.9’s persistence direction and the primary obligation lives in those other RFCs.

5.1 Hazards

Scenario H1.1 — Semantically distinct templates do not silently merge

• Given a corpus containing user logged in <*> and user logged out <*>
• When similarity threshold is 0.7 (the default)
• Then the two remain distinct template_ids
• And any widening produces an audit event recording both old and new templates

Scenario H1.2 — Lossy-zone match retains body

• Given a line whose best match has confidence in the lossy zone (floor ≤ x < threshold)
• When the line is ingested
• Then the body column contains the original line bytes
• And the row carries lossy_flag = false (the flag is reserved for tokenizer / preprocessing failure per §6.6 — the lossy zone retains the body but reconstruction still succeeds)

Scenario H1.3 — Every widening emits an audit event

• Given any sequence of inputs that triggers a template widening
• When the widening completes
• Then an audit event exists naming the old template, the new template, the tenant id, the timestamp, and the event_type

Scenario H1.4 — severity_number is part of the template key (no INFO/ERROR silent merge)

• Given two OtlpLogRecords with identical body_kind = String bodies and identical scope_name, but severity_number = 9 (INFO) and severity_number = 17 (ERROR)
• When both are ingested via MinerCluster::ingest
• Then the emitted records carry distinct template_ids
• And no widening or merge ever produces a single template_id covering both severity buckets
• (Operationalises the §6.1 Template-key composition commitment that severity_number is part of the key regardless of body_kind.)

Scenario H1.5 — scope_name is part of the template key (no cross-scope silent merge)

• Given two OtlpLogRecords with identical body_kind = String bodies and identical severity_number, but scope_name = Some("lib.auth") and scope_name = Some("lib.payments")
• When both are ingested
• Then the emitted records carry distinct template_ids
• And no widening or merge ever produces a single template_id covering both scopes
• And a third record with scope_name = None shares a template_id with neither (it lives in the (severity, None) bucket per §6.1)

Scenario H2.1 — Oversized parameter triggers OVERFLOW marker and forced body retention

• Given a tenant configured with the default 256 B per-parameter byte limit
• And a log line whose masked parameter value exceeds 256 B (e.g. an embedded stack trace)
• When the line is ingested
• Then the corresponding Param entry has type_tag = OVERFLOW carrying (length, sha256_prefix) instead of the original value
• And the body column contains the original line bytes regardless of lossy_flag
• And ourios.miner.params.overflow (attributes ourios.tenant, ourios.service) increments

Scenario H2.2 — Per-service overflow rate above 1% raises an alert

• Given the ourios.miner.params.overflow.utilization gauge (attributes ourios.tenant, ourios.service) for some service
• When the rolling rate exceeds 0.01
• Then the documented alert rule fires (the rule ships alongside §6.5’s metric definition)

Scenario H5.1 — Wildcard widening increments template_version and emits template_widened

• Given a leaf at (template_id = X, template_version = V)
• When an attach widens a previously-fixed token at position i into <*>
• Then the leaf’s template_version becomes V + 1
• And an audit event with event_type = template_widened is emitted naming the new wildcard position(s)

Scenario H5.2 — Type expansion increments template_version and emits template_type_expanded

• Given a leaf whose wildcard slot s has slot_types[s] = {NUM}
• When an attach maps a typed parameter of type = STR into slot s
• Then slot_types[s] becomes {NUM, STR}
• And template_version increments
• And an audit event with event_type = template_type_expanded is emitted naming the slot and the newly-added ParamType

Scenario H5.3 — Drift query returns templates that gained a version in window

• Given the template_audit event stream contains template_widened and template_type_expanded events for templates A and B in the window [t1, t2]
• When the §6.7 drift query runs against [t1, t2]
• Then the result includes both A and B with their widening counts

Scenario H7.1 — Reconstruction property holds across the corpus

• Given the committed testdata/corpus/ (anonymised, fixed)
• When every line is ingested through the miner
• Then for every emitted record r where r.lossy_flag = false, reconstruct(r) == r.ingested_bytes holds byte-for-byte
• And property failure is a build break, not a regression

Scenario H7.2 — Tokenizer failure sets lossy_flag = true and retains body

• Given a line containing an embedded NUL byte (or another tokenizer-failure mode listed in §6.6)
• When the line is ingested
• Then a parse-failure record is emitted
• And the record’s lossy_flag is true
• And the record’s body column contains the original line bytes

Scenario H7.3 — Reader emits body verbatim when lossy_flag is true

• Given a record with lossy_flag = true
• When the reader renders the row
• Then the rendered bytes are the body column verbatim (byte-for-byte, no prefix or in-band marker)
• And the rendered row carries the §6.6 reconstruction signal Reconstruction::RetainedVerbatim (the out-of-band warning marker the §6.6 Reader render contract defines)
• And reconstruct() is NOT called for that row

Scenario H7.4 — Widened literal slot reconstructs via STR fallback

• Given a leaf whose template gains a new <*> slot at position i via the §6.2 widening of an originally-literal token
• When the triggering line is attached
• Then the line’s record carries params[slot_for_i] = { type_tag: STR, value: L_tok[i] }
• And reconstruct(record) == ingested_bytes holds

5.2 Invariants

Scenario §3.1.1 — Default similarity threshold is 0.7

• Given a tenant configuration with no threshold override
• When the miner is initialised for that tenant
• Then the effective threshold is 0.7

Scenario §3.1.2 — Mandatory metric set is exposed

• Given the mandatory set defined by §6.8’s table — the ourios.miner.* metric instruments in the semconv registry (semconv/registry/, surfaced as generated ourios_semconv constants) that the miner registers on the ourios.miner meter when it is constructed (the ourios.miner.alias.* counters are registered separately by the alias map, §6.7, and are out of this scenario’s scope)
• When a small representative workload exercises every instrument (a normal line, a near-duplicate that widens a template, an oversized-param line, and a parse-failure line) and the meter is collected via an SDK in-memory reader
• Then the collected metric stream contains every metric named in §6.8’s table — each appearing on its first real measurement, carrying the registry’s required attributes (no synthetic zero-traffic points) — (ourios.miner.template.count, ourios.miner.merges, ourios.miner.confidence, ourios.miner.confidence.p50, ourios.miner.confidence.p01, ourios.miner.body_retention.utilization, ourios.miner.parse_failures, ourios.miner.params.overflow, ourios.miner.params.overflow.utilization, ourios.miner.template.version_changes, ourios.miner.duration) with the instrument kinds and attributes listed there

Scenario §3.2.1 — Default per-parameter byte limit is 256

• Given a tenant configuration with no per-parameter byte limit override
• When the miner is initialised for that tenant
• Then the effective limit is 256 bytes

Scenario §3.2.2 — Configured limit above 1 KiB is rejected at startup

• Given a tenant configuration with param_byte_limit > 1024
• When the miner is initialised
• Then initialisation fails with an error citing the §3.2 ceiling
• And the process refuses to start serving that tenant

Scenario §3.3.1 — Separators array captured on every successful tokenization

• Given a line that tokenizes successfully
• When the line is ingested
• Then the emitted record’s separators.len() == tokens.len() + 1
• And the per-row precondition for H7.1 holds (the reconstruction proptest then asserts byte equality)

Scenario §3.5.1 — Snapshot format carries a leading version byte

• Given a serialised snapshot artefact written by the miner
• When the artefact is inspected
• Then byte 0 is the snapshot format version

Scenario §3.5.2 — Unknown snapshot version triggers full WAL replay

• Given a snapshot artefact whose leading version byte is unknown to the running miner
• When the miner loads the snapshot at startup
• Then the snapshot is rejected
• And the miner falls back to full WAL replay rather than misinterpreting the bytes

Scenario §3.5.3 — Known-version restore + tail replay is equivalent to a full rebuild (2026-06-12 amendment)

• Given a tenant tree snapshotted at WAL high-water mark S, with further frames appended after S
• When recovery restores the snapshot and replays only the frames above S
• Then the recovered tree state (leaves, template_ids, template_versions, slot types, structured-template-id map) equals the control tree built by ingesting every record from scratch
• And no frame at or below S reaches the miner (no double-apply — the v1 hazard that gated restore)

Scenario §3.5.4 — Stale snapshot degrades loudly, not silently (2026-06-12 amendment)

• Given a snapshot at high-water mark S, a Parquet checkpoint at X > S, and a WAL whose surviving segments start above S but retain every frame above X (externally truncated — WAL segment files manually unlinked; the RFC 0008 §6.7 retain floor prevents this arising internally, and legitimate housekeeping never removes a frame above X)
• When recovery runs
• Then the snapshot is restored and the surviving frames are replayed under the per-consumer horizons (the data side is complete: every missing frame was ≤ X, hence already in Parquet)
• And a structured warning is emitted naming the gap between S and the oldest surviving frame, so the possible template re-minting inside it is surfaced (hazard #5, observable via the RFC 0010 drift query) rather than silent

Scenario §3.7.1 — Tenants’ template trees never cross-pollinate

• Given a MinerCluster ingesting interleaved lines from synthetic tenants A and B
• When the corpus is fully ingested
• Then no template mined under tenant A appears in tenant B’s tree
• And no template mined under tenant B appears in tenant A’s tree
• (Implements docs/benchmarks.md E2.)

Scenario §3.7.2 — Same structural template in two tenants gets distinct template_ids

• Given tenants A and B independently emit the structurally identical template user <NUM> logged in from <IP>
• When both are ingested
• Then tenant A’s template_id for that template differs from tenant B’s template_id
• And no template_id is shared across tenants
• And template_ids are guaranteed unique across the entire cluster (not just per tenant)

Scenario §3.7.3 — Tenant derivation runs per ResourceLogs, not per export batch

• Given a single OTLP ExportLogsServiceRequest carrying two ResourceLogs whose Resource.attributes resolve to distinct tenants A and B under the configured derivation rule
• When the receiver fans the batch out per RFC 0003 §6.3 and the miner ingests both per-tenant streams
• Then every LogRecord under ResourceLogs[0] is mined under tenant A
• And every LogRecord under ResourceLogs[1] is mined under tenant B
• And no record ever appears in the wrong tenant’s tree
• (Operationalises the §6.1 Tenant derivation commitment that the derivation rule runs once per inherited Resource, not once per export batch.)

5.3 RFC-internal design commitments

Scenario RFC0001.1 — Fresh-leaf creation does not emit an audit event

• Given a parent prefix node with no leaves yet
• When a line creates the first leaf at that node
• Then no event is appended to the audit stream for that creation
• And ourios.miner.template.count increments to reflect the new leaf

Scenario RFC0001.2 — Degenerate-template guard rejects fully-wildcard widening

• Given a leaf whose template, after a candidate widening, would have zero non-wildcard tokens
• When the candidate widening is attempted
• Then the widening is rejected
• And the line is treated as a parse failure (confidence = 0, body retained, ourios.miner.parse_failures increments)
• And an audit event with event_type = template_widening_rejected_degenerate records the rejection

Scenario RFC0001.3 — Tokenizer is Unicode whitespace only; punctuation stays in tokens

• Given a line key=value, other=42 (no whitespace adjacent to the punctuation)
• When the line is tokenized
• Then it produces two tokens (key=value, and other=42)
• And no token boundary is introduced at =, ,, :, ;, [, ], (, or )

Scenario RFC0001.4 — Confidence ratio = simSeq / threshold; decision boundary at 1.0

• Given a tenant with threshold = 0.7
• And a line whose simSeq against the best candidate is 0.7
• When the line is ingested
• Then the emitted record’s confidence == 1.0
• And the line takes the clean-attach branch
• And the same simSeq under threshold = 0.5 would yield confidence == 1.4 (the ratio reframes scale-invariantly across tenants)

Scenario RFC0001.5 — Bare template_id = X spans all versions of leaf X

• Given leaf X with versions 1, 2, 3 attached over time
• When a query runs where template_id = X
• Then the result includes rows attached against (X, 1), (X, 2), and (X, 3)
• And no alias resolution is involved (this is by-construction, since template_id is stable across widenings of one leaf)

Scenario RFC0001.6 — Bare template_id = X does NOT follow alias chains

• Given two distinct leaves X and Y that the alias index records as semantically equivalent
• When a query runs where template_id = X
• Then only rows whose template_id == X are returned; rows with template_id == Y are NOT included
• And where template_id.resolves_to(X) (RFC 0002 §5.4) is the explicit form that includes Y’s rows

Scenario RFC0001.7 — Combined widening + type-expansion increments version twice and emits two events in order

• Given a leaf at version V where a single attach both introduces a new wildcard slot AND introduces a previously-unseen ParamType into an existing slot
• When the attach completes
• Then the leaf’s template_version == V + 2
• And the audit stream contains two events for this attach: a template_widened event for the new wildcard, immediately followed by a template_type_expanded event for the type expansion (in that order)

Scenario RFC0001.8 — ourios.miner.confidence.p50 and ourios.miner.confidence.p01 are emitted as gauges

• Given a running miner with a non-empty ourios.miner.confidence histogram for some (ourios.tenant, ourios.service)
• When the miner’s meter is collected via an SDK in-memory reader
• Then ourios.miner.confidence.p50 and ourios.miner.confidence.p01 (attributes ourios.tenant, ourios.service) are present as gauges
• And each value matches the corresponding quantile of the same-attributed histogram (computed in-process on a short ticker per §6.8)

(The dotted-semconv rename landed in the 2026-06-08 amendment; the open fork is whether confidence.p50 / confidence.p01 become backend-derived quantiles over the exported histogram rather than in-process gauges. That is a contract change to the §3.1.2 mandatory set and would be made — possibly superseding this scenario — under its own review, not folded into the rename.)

Scenario RFC0001.9 — body_kind = Structured short-circuits to a structured-template id

• Given an OtlpLogRecord whose body is Body::Structured(AnyValue) (any non-String AnyValue variant carried verbatim per RFC 0003 §6.4)
• When the record is ingested
• Then the §6.2 algorithm skips tokenize/mask/descend per step 0 and allocates or reuses the structured-template id for (severity_number, scope_name, BodyKind::Structured)
• And the emitted record has body_kind = Structured
• And the emitted record’s body carries the Ourios canonical body encoding of that AnyValue (per the §6.1 encoding rule)
• And params and separators are empty
• And confidence == 1.0 (the §6.1 sentinel)
• And lossy_flag == false

Scenario RFC0001.10 — time_unix_nano is preserved verbatim from the wire

• Given an OtlpLogRecord with time_unix_nano = 1_715_700_000_000_000_000
• When the record is ingested and committed to Parquet
• Then the emitted row has time_unix_nano == 1_715_700_000_000_000_000
• And a query WHERE time_unix_nano BETWEEN 1_715_600_000_000_000_000 AND 1_715_800_000_000_000_000 returns the row
• (Gates docs/benchmarks.md B1 — time-range queries — by making the underlying column measurable.)

Scenario RFC0001.11 — severity_number = 0 and scope_name = None are distinct key buckets

• Given four OtlpLogRecords with identical body_kind = String body, varying only in (severity_number, scope_name) across (0, None), (0, Some("lib.x")), (9, None), (9, Some("lib.x"))
• When all four are ingested
• Then four distinct template_ids are emitted, one per key bucket
• And no widening or merge ever coalesces the severity_number = 0 (UNSPECIFIED) bucket with any specified-severity bucket
• And no widening or merge ever coalesces the scope_name = None bucket with any scope_name = Some(_) bucket
• (Locks the §6.1 explicit edge-case rules: 0 = UNSPECIFIED is a valid OTLP severity that gets its own bucket, and absent scope is its own bucket.)

Scenario RFC0001.12 — Alias assertion is durably recorded and appears in the per-tenant map

• Given tenant T with two distinct leaves A and B (A < B) and no existing alias set
• When an operator asserts B is an alias of A
• Then an alias_asserted audit event is durably recorded on the §6.4 stream under the §3.4 WAL-before-ack barrier before the assertion is acknowledged, naming tenant_id = T, the anchor representative_id = A, member_ids = [B], the actor, and the timestamp — so the asserted set is {A} ∪ {B} = {A, B}
• And after the per-tenant projection rebuilds, tenant T’s alias map contains an equivalence class with members {A, B} whose derived canonical representative is A (the smallest member)

Scenario RFC0001.13 — resolves_to(rep) returns all members and excludes non-members

• Given tenant T whose alias map records the set {A, B} (per RFC0001.12) and an unrelated leaf C in no set
• When the querier compiles template_id.resolves_to(A) for tenant T
• Then the predicate expands to template_id IN {A, B}
• And resolves_to(B) expands to the same {A, B} (expansion is by the set, not the direction of assertion)
• And resolves_to(C) expands to exactly {C}

Scenario RFC0001.14 — Cross-tenant isolation: an alias in tenant A never affects tenant B [§3.7]

• Given tenant T1 whose alias map records {A, B} and tenant T2 that has the same template_ids A and B but no alias assertion
• When the querier compiles template_id.resolves_to(A) once for T1 and once for T2
• Then for T1 it expands to {A, B}
• And for T2 it expands to exactly {A}
• (Locks §3.7: alias sets are per-tenant; an assertion in one tenant is invisible to every other.)

Scenario RFC0001.15 — Retraction removes any member, including the canonical, and is itself audited

• Given tenant T whose alias map records the equivalence class {A, B} (A < B, so A is the derived canonical)
• When an operator retracts member A — the canonical / smallest member — from the class
• Then an alias_retracted audit event is durably recorded (same WAL-before-ack barrier and field shape as RFC0001.12) whose asserted set names A (here as representative_id, the operator’s anchor) plus an empty member_ids, and the actor
• And after the projection rebuilds, A is removed from the class, leaving {B} — a single member, which is no longer an alias set, so resolves_to(A) expands to exactly {A} and resolves_to(B) expands to exactly {B}
• (Locks the representative-independent retraction rule: retracting any member is well-defined even when it is the canonical/smallest; the canonical is re-derived as min of the remainder, and a class that drops to one member ceases to be an alias set.)

Scenario RFC0001.16 — A non-aliased id resolves to itself

• Given tenant T with leaf Z and no alias assertion naming Z
• When the querier compiles template_id.resolves_to(Z)
• Then the predicate expands to exactly {Z} — identical to the base-member behaviour and to bare template_id = Z (RFC0001.6)

6. Proposed design

The Ourios miner in detail. This is the section that the ourios-miner crate is implemented against; §5’s Acceptance criteria operationalise the commitments here, and §8 maps each §5 scenario to the technique that tests it.

Why §6 exists. Published Drain (§3) and Drain3 (§4) do not address the properties Ourios requires. Each row below is a gap this section closes:

6.1 Data model

Amendment 2026-05-13. Section rewritten to align the record schema with the OTLP LogRecord shape — the project’s stated ingest contract per docs/glossary.md (entry OTLP: “we do not invent our own format”). The investigation that surfaced the gap is docs/architecture/otlp-log-format.md. The pre-amendment schema treated logs as raw text strings; the amended schema treats every log as a structured OTLP record from the moment it enters the system. §6.2’s algorithm and its ingest signature were aligned to this amendment in a companion edit the same day (see §6.2’s amendment note below): the body.kind fork is at the top of the algorithm, the descent step incorporates the §6.1 template-key tuple, and the MinerCluster::ingest signature now takes a structured OtlpLogRecord rather than a raw &str.

The miner emits one record per ingested OTLP LogRecord. The record shape mirrors the wire shape of OTLP logs (the opentelemetry-proto LogRecord plus its inherited Resource and InstrumentationScope context) plus the miner-derived columns that template mining produces.

Record columns

The record carries three groups of columns. The OTLP-derived group preserves the structured shape the wire promised; the miner-derived group is what this RFC introduces; the reconstruction group exists only when the body was mineable (body.kind = String).

Identity and partitioning:

OTLP-derived columns (faithful to opentelemetry-proto):

Amendment 2026-06-11 — the effective timestamp lives in RFC 0005, not here. RFC 0005 §3.2 (amendment of the same date) adds a writer-derived effective_time_unix_nano Parquet column — time_unix_nano when non-zero, else observed_time_unix_nano, else 0 — following the OTLP logs data model’s recommendation (“Use Timestamp if it is present, otherwise use ObservedTimestamp”). The record shape above is unchanged: the miner emits no new field, the Parquet writer computes the column from the two timestamp fields already listed, and the wire time_unix_nano is stored verbatim including 0 — scenario RFC0001.10 (verbatim preservation) remains intact and normative. Time partitioning and the DSL time window key off the derived column (RFC 0005 §3.4 / RFC 0002 §6.2).

Body and miner-derived reconstruction:

Where:

• Param = { type_tag: ParamType, value: Bytes }. ParamType is one of IP, UUID, NUM, HEX, TS, PATH, STR, OVERFLOW. STR is the unmasked-wildcard fallback — used when a slot was created by template widening of a previously-fixed literal token (the literal itself becomes the param value); OVERFLOW carries (length: u32, sha256_prefix: [u8; 8]) instead of the original value (§6.5). params.len() == count(<*> in template), always (in the body_kind = String branch); §6.2 enforces this when a widening introduces new wildcard slots.
• Separator is a small inline byte string (typically 1–3 bytes in practice). Encoding in Parquet is an implementation detail that does not affect this RFC.
• KeyValue mirrors the OTLP KeyValue message: a key: String and a value: AnyValue. AnyValue is a discriminated union over string | bool | int | double | bytes | array | kvlist. Storing AnyValue faithfully in Parquet (rather than flattening to a string) is what keeps query expressions like attributes["client.address"] = "10.0.0.1" typed.
• BodyKind is a two-variant enum (String, Structured) — not the full AnyValue discriminator. The body column carries the encoded AnyValue payload; body_kind is the cheap routing flag the query planner uses to decide whether reconstruction is defined for this row.

Body representation (AnyValue handling)

OTLP’s LogRecord.body is AnyValue — string, bool, int, double, bytes, array, or kvlist. The spec is explicit (Logs Data Model §Body): “Body MUST support AnyValue to preserve the semantics of structured logs emitted by the applications.” Real OTel emitters send structured Body routinely, not just text.

Ourios distinguishes two body shapes at ingest:

• body_kind = String — LogRecord.body is AnyValue::String. The miner runs the §6.2 algorithm over the unwrapped string: tokenize, mask, descend the tree, attach to or create a leaf. params, separators, confidence, lossy_flag are populated per the existing semantics.
• body_kind = Structured — LogRecord.body is any other AnyValue variant (kvlist, array, int, double, bool, bytes). The miner does not run the §6.2 algorithm. The body is encoded with the Ourios canonical body encoding (see The Ourios canonical body encoding below) and stored in the body column; no template is mined, no params/separators are emitted. template_id is allocated per the Template-key composition rule below — for this branch the key is (severity_number, scope_name, BodyKind::Structured), so all structured-Body records sharing a (severity, scope) share one template_id. The leaf the id points at carries the Structured marker and an empty body_template. confidence = 1.0 (sentinel), lossy_flag = false (the canonically-encoded body is authoritative; nothing is reconstructed from a template).

This is the conservative default. It preserves the structural content of the body (the Ourios canonical body encoding below makes [§3.3] reconstruction well-defined for the structured branch: stored_bytes ↔ AnyValue is bidirectional and byte-deterministic), it avoids inventing template structure for arbitrary AnyValue trees, and it sidesteps the spec ambiguity of “what is the template for {"msg": "x", "user_id": 42}.” A future opt-in mine-inner-field mode (e.g., mine body.kvlist["msg"] as the line if present) is a configurable knob, not the default; that decision lives with the maturity-stage move from red → green once corpus evidence informs which inner-field conventions are worth specifying.

A third path — render-to-string + mine (canonicalise structured Body to JSON-ish text and run it through the §6.2 mining algorithm) — was rejected because mining over the JSON serialisation produces token templates that depend on the serialiser’s whitespace and field-ordering choices, which is both fragile (changing serialisers shifts every template) and defeats the §3.3 reconstruction guarantee for any record where the original wire form was protobuf rather than JSON. Storing the canonical encoding (without mining over it) is different from this rejected path: storage is faithful, it just doesn’t get a template extracted.

The Ourios canonical body encoding (body_kind = Structured).

Amendment 2026-06-09 (no canonical OTLP JSON exists). This paragraph previously called the encoding “the OTLP-canonical JSON encoding per the OTLP specification’s HTTP/JSON binding,” implying a spec-defined canonical form. There is none. Per the OTLP spec, the OTel common docs, and a maintainer answer (Josh Suereth, 2026-06-09): OTLP/JSON is the proto3 JSON mapping plus a short closed list of deviations (hex trace_id/span_id, integer enums, ignore-unknown-fields, lowerCamelCase) — with no normative rules on whitespace, key/field ordering, or number canonicalisation, and OTLP does not require lossless translation between formats. The text below is reframed to state the rule as an Ourios-local deterministic encoding, not an OTLP conformance point, and renamed to “the Ourios canonical body encoding.” No code and no RFC status change here; the encoder is ourios-core’s otlp::canonical (a separate follow-up PR aligns its doc comments).

Amendment 2026-06-11 (doubles round-trip bit-exactly — #130). The rule below previously left double precision implicit, and decode(encode(x)) drifted 1–2 ULP for ~12% of arbitrary finite f64. Investigating #130 located the loss on the decode side, not the encoder: the emitter already produces shortest-round-trip digits (serde_json’s Ryu f64 formatter — with-serde adds no custom double formatter), but serde_json’s default float parsing is approximate. The rule now pins both halves: doubles are emitted as shortest-round-trip JSON numbers and decoded with correctly-rounded float parsing (serde_json’s float_roundtrip feature, declared load-bearing in ourios-core), so the f64 round-trip is exact for every finite double — the faithfulness guarantee below holds for arbitrary doubles, not just “nice” ones. Non-finite doubles (NaN, ±∞) have no JSON-number form and encode as null — bytes that do not decode back; that pre-existing gap is explicitly out of scope here and stays open. No RFC status change.

The body column carries the Ourios canonical body encoding of the AnyValue: a proto3-JSON form, defined below. This is an Ourios-local deterministic convention, not an OTLP-mandated canonical form. OTLP defines no canonical or byte-deterministic JSON encoding. Its only normative JSON rules are the proto3 JSON mapping (per the protobuf spec) plus a short closed list of OTLP-specific deviations — trace_id / span_id as hex strings (not applicable to a body AnyValue, which carries no IDs), enum values as integers, ignore unknown fields, and lowerCamelCase field names. The spec is silent on whitespace, key/field ordering, and number canonicalisation, and OTLP does not require lossless translation between formats. There is therefore no “canonical OTLP JSON” to reference; the byte-stable encoding below is Ourios’s own, chosen so the body column is byte-deterministic (storage dedup) and the [§3.3] reconstruction guarantee is well-defined.

The concrete rule is the proto3 JSON mapping as emitted by opentelemetry-proto’s with-serde feature via serde_json:

• field names in lowerCamelCase;
• int64 / uint64 values as decimal strings (proto3 JSON’s canonical emit form; decoders accept a JSON number or string);
• double values as JSON numbers in shortest-round-trip form, decoded with correctly-rounded float parsing — decode(encode(x)) is bit-exact for every finite f64 (#130; see the 2026-06-11 amendment above);
• bytes as base64;
• KvlistValue and ArrayValue element order preserved as received — not sorted (this is explicitly not RFC 8785 / JCS canonical JSON);
• byte-deterministic because the proto types have a fixed serde field order and serde_json serialisation is deterministic.

Canonical byte examples: {"intValue":"-42"}, {"doubleValue":2.71}, {"boolValue":true}, {"bytesValue":"<base64>"}, {"arrayValue":{"values":[…]}}, {"kvlistValue":{"values":[{"key":"…","value":{…}}]}}.

“Deterministic” here means byte-identical. Re-encoding the same in-memory AnyValue with this encoder yields byte-for-byte identical output (within a fixed opentelemetry-proto / serde_json version). This is the byte-level reading, not a weaker struct-level one: it is what lets the body column be deduplicated and lets two receivers handling the same logical AnyValue produce the same stored bytes. The receiver path: OTLP/gRPC (protobuf wire) decodes to an in-memory AnyValue and re-encodes here; OTLP/HTTP+JSON decodes the incoming JSON to an in-memory AnyValue and re-encodes the same way, so the stored bytes do not depend on the producer’s whitespace, field order, or int64-as-number-vs-string choice.

The faithfulness guarantee — stored_bytes decode back to the original in-memory AnyValue — is an Ourios guarantee delivered by this encoder/decoder pair, not an OTLP lossless promise (OTLP makes none). lossy_flag = false for structured rows rests on this Ourios guarantee, not on any OTLP conformance claim.

Duplicate keys. OTLP KvlistValue is a repeated KeyValue that the data model treats as a map with unique keys; the data-model map equality is order-insensitive, but OTLP does not define wire-order equality, which is why preserving received order (above) is the safe, spec-permitted choice. A KvlistValue carrying duplicate keys is non-conforming OTLP input with no defined semantics. Ourios does not silently dedup or reorder such input: it preserves the entries verbatim in the encoding (so the round-trip stays faithful) and makes no map-semantic guarantee for it, rather than inventing one.

Template-key composition

A template’s identity (the discriminator the Drain tree uses to decide “is this the same template?”) depends on the body shape:

• body_kind = String — key tuple is (severity_number, scope_name, masked_body_tokens).
• body_kind = Structured — key tuple is (severity_number, scope_name, BodyKind::Structured). All structured-Body records sharing a (severity_number, scope_name) share one template_id. This intentionally forfeits structured-body shape clustering — the rationale is that the structured-Body branch’s value comes from the faithful preservation of attributes and the canonically-encoded body, not from grouping similar AnyValue shapes. Operators who need shape-level clustering can opt into a future body_shape_fingerprint column (a stable hash over the AnyValue’s structural skeleton — kvlist key-set, nested shape, leaf-type sequence; values ignored) as a reserved extension; the gate for adding it is “we have a concrete consumer,” not “it might be useful.”

The bullet rationale below applies to both branches:

• severity_number is part of the key because INFO and ERROR versions of the same body text are semantically distinct events. “user logged in” at INFO is a routine signal; “user logged in” at ERROR is an alarm (or an emitter bug) — collapsing them to one template_id would surface either as the other on query, which is a [§3.1] “no silent merges” violation in disguise. The OTLP-spec-valid severity_number = 0 (UNSPECIFIED) is a distinct key value, not coalesced with any specified severity.
• scope_name is part of the key because the same body text emitted from two different instrumentation scopes (myapp.login vs myapp.checkout) describes two different events. The scope is the OTel-canonical “which code path emitted this,” directly analogous to the package/logger name in traditional logging frameworks. Records with no scope (scope_name = None) cluster as their own (severity, None) bucket.
• resource_attributes are NOT part of the key. They identify who sent the record (service, host, k8s pod), not what event was emitted. The tenant_id derivation (below) already encodes the partition decision over Resource. Folding Resource into the template key would explode template cardinality proportionally to the deployment fleet size without adding semantic discrimination — the same myapp.login template from two replicas of service.name = api is the same template.
• event_name is not in the key today but is reserved as a candidate addition. RFC 0001 stays at the OTLP-canonical severity+scope key; promoting event_name into the key is a follow-up RFC patch once corpus evidence justifies it.

The Drain tree’s implementation of this tuple (extra prefix levels above length-N, tuple-keyed leaf lists, separate trees per (severity, scope), etc.) is §6.2 implementation territory and may be revisited based on cardinality observations from the corpus benchmark. The RFC pins only the semantic key.

Tenant derivation

tenant_id is derived per ResourceLogs group, not per OTLP export batch. Each ResourceLogs carries its own Resource.attributes, and a single OTLP export can contain multiple ResourceLogs groups from different sources — so one export can route records to multiple tenants. The derivation runs once per inherited Resource; the resulting tenant_id applies to every LogRecord under that ResourceLogs group (across all its ScopeLogs), and the receiver fans the records out into per-tenant streams.

The default per-Resource rule:

tenant_id := resource.attributes["service.name"]   if present
          ?: <operator-required fallback rule>

service.name is the conventional OTel unit of “what application emitted this,” and it maps directly onto Ourios’s per-tenant template-tree partitioning ([§3.7]). Operators with a different multi-tenant model (per-namespace, per-customer-id-attribute, composite of multiple attributes) configure an alternative rule; the receiver does not invent a tenant identity that the operator hasn’t declared.

If a ResourceLogs group’s Resource resolves to no tenant under either the default rule or the operator’s fallback, the receiver rejects the entire export batch with a controlled error (no panic, no silent assignment to a “default” tenant; the sender sees the failure and either fixes its emitter or its deployment). Per-Resource rejection within an otherwise-valid batch is not supported in this RFC — the all-or-nothing failure mode is simpler to reason about for the sender, and OTLP’s batch-level acknowledgement model fits all-or-nothing more naturally than partial-success. The receiver-side specification of this rejection path (and any future opt-in for partial acceptance) lives in RFC 0003 — OTLP receiver (forthcoming).

Template identity

template_id is a cluster-wide unique monotonic u64 (with each tenant seeing a monotonic subsequence), allocated when a new leaf is created and never reused or reassigned. The id space is shared across tenants so that the same u64 value never refers to two different leaves; the per-tenant subsequence guarantee preserves [§3.7] by making each tenant’s allocation order observable in isolation. Cross-tenant content identity is intentionally not guaranteed — two tenants emitting the structurally identical template (same (severity_number, scope_name, masked_body_tokens) tuple) will have different template_ids, so a template_id alone never links structurally-equivalent templates across tenants. (The u64 value itself is cluster-wide unique, per the previous paragraph; what is not guaranteed is that the same template across two tenants resolves to the same id.) This preserves [§3.7] (per-tenant template trees) by construction; cross-tenant analytics that need content identity (deduplication across tenants for storage savings, shared template dashboards) are an opt-in concern and are not provided by the miner. A future template_fingerprint side column may carry a canonical content hash over (severity_number, scope_name, masked_body_tokens) for opt-in cross-tenant use; the gate for adding it is “we have a concrete consumer,” not “it might be useful.”

Template version. template_version starts at 1 when the template is created and increments by 1 on every widening event: either a new wildcard slot opens (a previously fixed token at position i becomes <*>), or an existing wildcard’s typed parameter set changes (e.g. a <NUM> slot starts seeing <STR> values). To detect the second case, every leaf carries — alongside its template — a slot_types: Vec<HashSet<ParamType>> indexed by wildcard slot, recording every ParamType observed in that slot. A type expansion is the addition of a ParamType to one of these sets. The pair (template_id, template_version) uniquely identifies one structural state of a template. Queries against template_id = X return all versions; queries against (template_id, template_version) = (X, V) return only the named state. The DSL surface is RFC 0002’s concern, not this RFC’s, but the data model must support both.

Why two integers and not a content hash. A content hash makes identity global by construction; in a multi-tenant backend that is a tenant-isolation leak rather than a feature. A content hash also makes template_version redundant — once the canonical template string changes, the hash changes, so versioning collapses into alias-mapping between hashes. Per-tenant monotonic ints with an explicit version field are smaller in the Parquet column, easier to reason about under [§3.7], and keep (template_id, template_version) as a meaningful compound key.

6.2 Algorithm

Amendment 2026-05-13. Rewritten to take a structured OTLP LogRecord rather than a raw &str, in line with the §6.1 amendment. The algorithm now opens with the body.kind fork from §6.1’s Body representation: AnyValue::String runs the Drain mining steps (the prior algorithm, preserved verbatim below); every other AnyValue variant short-circuits to the structured emit per §6.1’s Template-key composition fork. Step 3’s descent now incorporates (severity_number, scope_name) into the tree key, again per §6.1 — the implementation choice (extra prefix layers, tuple-keyed leaf lists, separate trees per (severity, scope)) stays in §6.2 as the algorithm’s responsibility, but the semantic key is pinned by §6.1. The ingest signature on MinerCluster becomes ingest(record: &OtlpLogRecord); pre-amendment callers were ingest(tenant_id, raw: &str).

The miner sees an already-tenant-resolved (tenant_id, record: OtlpLogRecord) pair. The receiver (RFC 0003) is responsible for resolving tenant_id per ResourceLogs and fanning records into per-tenant streams before the miner sees them; §6.1’s Tenant derivation pins that contract.

For each ingested OTLP LogRecord:

0.  match record.body.kind:

      AnyValue::String(s):
          # Continue with the Drain mining algorithm in steps
          # 1–5 below, treating `s` as the `L_raw` of the prior
          # spec. body_kind = String.

      AnyValue::Bool | Int | Double | Bytes | Array | KVList:
          # Structured short-circuit per §6.1 *Body
          # representation*. The miner does NOT run the Drain
          # mining steps. body_kind = Structured.
          encoded = encode_canonical_body(record.body)
              # Ourios canonical body encoding (a proto3-JSON form;
              # lowerCamelCase fields, int64/uint64 as decimal
              # strings, bytes as base64, kvlist/array order
              # preserved — NOT sorted). This is an Ourios-local
              # deterministic convention, NOT an OTLP-mandated
              # canonical form: OTLP defines no canonical JSON and
              # requires no lossless translation. For records over
              # OTLP/gRPC the receiver decodes protobuf and
              # re-encodes here; for OTLP/HTTP+JSON it decodes to
              # the in-memory AnyValue and re-encodes the same way,
              # so stored bytes are byte-identical regardless of the
              # producer's whitespace / field order / int64 form.
              # The lossy_flag = false promise rests on this Ourios
              # round-trip guarantee — see §6.1 for the why.
          template_id = allocate_or_reuse_structured_template_id(
              record.severity_number,
              record.scope_name,
          )
              # Per §6.1 *Template-key composition*, the
              # structured-Body key is (severity_number,
              # scope_name, BodyKind::Structured). All structured
              # records sharing a (severity, scope) share one
              # template_id. The leaf the id points at carries the
              # `Structured` marker and an empty body_template.
          attach_structured(record, encoded, template_id,
                            confidence = 1.0,
                            lossy_flag = false)
              # confidence = 1.0 sentinel; lossy_flag = false
              # because the canonicalised body is authoritative,
              # nothing is reconstructed from a template.
          return

1.  L_tok, separators = tokenize(L_raw)
        # tokenize splits on Unicode whitespace only — every
        # codepoint matching `char::is_whitespace()` (ASCII space,
        # tab, CR, LF, plus the broader Unicode whitespace classes
        # U+0085, U+00A0, U+1680, U+2000–U+200A, U+2028, U+2029,
        # U+202F, U+205F, U+3000). Every other byte (including
        # punctuation such as `=`, `:`, `,`, `;`, `[`, `]`, `(`,
        # `)`) stays inside a token; structured separators are the
        # masking layer's responsibility (§4.2 / step 2). The
        # captured whitespace runs go into `separators` so that
        # reconstruction (§6.6) is byte-identical.
        # On failure (malformed UTF-8, embedded NUL, line longer
        # than max-line-bytes): emit a parse-failure record and
        # increment ourios.miner.parse_failures. Skip the rest.
        # Note: an empty-after-whitespace string (the AnyValue
        # carries `""` or only whitespace) is not a parse failure
        # — it has zero tokens and the miner short-circuits with
        # the cluster's `NO_TEMPLATE` sentinel rather than
        # descending the tree. The pre-amendment cluster code
        # already routes this case; the spec just records it.

2.  L_masked, typed_params = mask(L_tok)
        # mask applies the configured masking rules in order;
        # any token matching a rule is replaced with its type
        # tag (e.g. <IP>) and the original bytes are pushed
        # into typed_params with that tag. Unmasked tokens
        # remain literal.

3.  parent = tree.descend(record.severity_number,
                           record.scope_name,
                           len(L_masked),
                           L_masked[0..d-1])
        # Per §6.1 *Template-key composition*, the discriminator
        # for "is this the same template?" is the tuple
        # (severity_number, scope_name, masked_body_tokens).
        # Step 3 incorporates severity_number and scope_name into
        # the descent key alongside the masked-token prefix used
        # by published Drain. The implementation may layer extra
        # prefix levels above the length-N node, key leaf lists
        # by (severity, scope), or maintain separate trees per
        # (severity, scope) — the choice is cardinality-driven
        # and revisitable from corpus observations. The
        # severity_number = 0 (UNSPECIFIED) and scope_name = None
        # cases are valid distinct key positions; they cluster as
        # their own buckets, never coalesced with any specified
        # severity or named scope.
        # if a node along the path does not exist, create it.

4.  candidate = argmax over leaf in parent.leaves of
                  simSeq(L_masked, leaf.template)
    if candidate is None:
        # no leaves under parent yet; create one. Creation does not
        # emit an audit event — `ourios.miner.template.count` already reflects
        # leaf allocation, and §6.4 reserves the audit stream for
        # widening events whose semantics need cross-referencing.
        leaf = new Leaf(template = L_masked)
        parent.leaves.push(leaf)
        # On fresh-leaf creation the template is L_masked verbatim,
        # so every <*> in it came from mask(); params == typed_params.
        attach(L_masked, typed_params, separators, leaf,
               confidence = 1.0, lossy_flag = false)
        return

5.  similarity = simSeq(L_masked, candidate.template)
    confidence = similarity / threshold

    if similarity >= threshold:
        # clean or lossy attach; widen the template if needed.
        # widen() returns:
        #   widened           — the new template (existing fixed
        #                       positions that mismatched L_masked
        #                       become <*>)
        #   new_wildcards     — the set of positions that just
        #                       became <*> (the audit payload)
        widened, new_wildcards = widen(candidate.template, L_masked)
        if new_wildcards > 0:
            candidate.template = widened
            candidate.version += 1
            emit_audit(event_type = template_widened,
                       template_id = candidate.id,
                       old_version, new_version = candidate.version,
                       positions_widened = new_wildcards.positions,
                       ...)
            ourios.miner.merges.inc()

        # Build the params array. One entry per <*> in the (possibly
        # just-widened) template, in template order. For each slot:
        #   - if the slot existed before this attach AND mask() emitted
        #     a typed_params entry for it, use that entry verbatim.
        #   - if the slot is a fresh wildcard from this widening (the
        #     position held a literal token in candidate.template before
        #     the widen call), the original literal at that position in
        #     L_tok is captured as { type_tag: STR, value: L_tok[pos] }.
        # Without this step §6.1's "one entry per <*> slot" invariant
        # is violated and §6.6's reconstruct() has no value to insert
        # at the freshly-widened position.
        params = build_params(candidate.template, typed_params,
                              L_tok, new_wildcards)

        # Type-expansion: if any wildcard slot now sees a typed param
        # whose type tag is not already in that slot's observed-type
        # set, widen the slot's type set, bump the version, and
        # audit. The leaf carries `slot_types: Vec<HashSet<ParamType>>`
        # alongside its template (data model in §6.1).
        new_types = update_slot_types(candidate, typed_params)
        if not new_types.is_empty():
            candidate.version += 1
            emit_audit(event_type = template_type_expanded,
                       template_id = candidate.id,
                       old_version, new_version = candidate.version,
                       slots_expanded = new_types,
                       ...)

        attach(L_masked, params, separators, candidate,
               confidence,
               lossy_flag = false)  # §6.6: lossy_flag is set only on
                                    # tokenizer/preprocessing failure,
                                    # not on confidence < 1.0
        return

    if similarity >= floor:
        # lossy zone: the line is "close" but doesn't meet
        # threshold. Create a new leaf rather than force-merging.
        # Body retention is unconditional in this branch.
        leaf = new Leaf(template = L_masked)
        parent.leaves.push(leaf)
        # As in the candidate-is-None branch, the new leaf's template
        # is L_masked verbatim, so params == typed_params.
        attach(L_masked, typed_params, separators, leaf,
               confidence,
               lossy_flag = false,
               body = L_raw)  # forced body retention
        ourios.miner.body_retention.utilization.observe(retained = true)
        return

    # similarity < floor: parse failure
    ourios.miner.parse_failures.inc()
    emit_failure_record(L_raw, reason = "no candidate above floor")

Branching invariants:

• Step 0’s structured short-circuit never enters the Drain mining steps (1–5). Structured-Body records do not widen, do not emit template_widened or template_type_expanded audit events, do not contribute to ourios.miner.merges, and never carry params/separators. The structured branch’s [§3.1] preservation is vacuous: no template merge happens, so no silent merge is possible.
• The tree only deepens on first observation of a (severity_number, scope_name, length, prefix tokens) shape (the §6.1 template-key tuple, anchored at this section’s step 3).
• Leaves are split (new leaf created) when the best candidate is in the lossy zone; they are never split when the candidate is clean.
• A leaf is widened (wildcards introduced) when a clean attach would otherwise mismatch positions. Every widening emits an audit event (§6.4).
• A leaf’s wildcard slot is type-expanded when an attach maps a typed parameter whose ParamType is not already in that slot’s slot_types[slot] set. Type expansion increments template_version and emits a template_type_expanded audit event (§6.4).
• A single attach can trigger both wildcard-widening and type-expansion in the same leaf; in that case template_version increments twice and two audit events are emitted, in that order.
• The leaf’s template_version only increments on widening or type-expansion, not on a clean attach. Structured-Body leaves are never widened or type-expanded; their template_version stays at 1 for the lifetime of the leaf.

6.3 Confidence scoring [§3.1]

confidence = simSeq / threshold. The ratio framing makes the decision boundary land at confidence == 1.0 regardless of the configured threshold, which gives ourios.miner.confidence.p50 and ourios.miner.confidence.p01 ([§3.1] required metrics) a stable interpretation across tenants with different thresholds: the p01 value tells you how close the bottom 1% of attaches are to the merge boundary. A collapsing p01 means many lines are barely passing — a tuning signal even though the threshold itself has not moved.

Three zones, with concrete defaults:

• confidence ≥ 1.0 (i.e. simSeq ≥ threshold): clean attach. No body retention.
• floor / threshold ≤ confidence < 1.0: lossy zone. The line attaches to a freshly created leaf rather than being force-merged into the candidate (see §6.2 step 5). body is retained unconditionally; lossy_flag follows the §6.6 rule (set only on reconstruction failure, not on lossy zone alone — the body is available either way).
• confidence < floor / threshold: parse failure. ourios.miner.parse_failures increments; the line is written to a failure record with the original bytes intact.

Defaults. threshold = 0.7, floor = 0.4. The threshold floor is fixed by [§3.1] (“threshold ≥ 0.7, lowering requires an RFC, not a config change”); the lossy-zone floor is a tuning knob between threshold and 0. floor = 0.4 matches the paper’s reported default threshold, on the reasoning that lines below the paper’s own bar are likely genuinely different events. Tuning the floor is a per-tenant config decision; it is not load-bearing for any invariant.

6.4 Merge policy [§3.1]

A template widening (per §6.2 step 5) is the operation that [§3.1] calls a “merge.” Every widening emits an audit event with the schema:

{
  event_type: AuditEventType,  # enum:
                               #   template_widened
                               #   template_type_expanded
                               #   template_widening_rejected_degenerate
  tenant_id: TenantId,
  template_id: u64,
  old_version: u32,
  new_version: u32,
  old_template: String,        # canonical form, with <*> for wildcards
  new_template: String,
  triggering_line_hash: [u8; 16],  # blake3 of L_raw, for cross-ref
  triggering_line_sample: Option<String>,  # first 256 B of L_raw
  positions_widened: Vec<u16>, # token positions that became <*>
                               # (empty for template_type_expanded)
  slots_expanded: Vec<SlotExpansion>,
                               # slot index + newly added ParamType(s)
                               # (empty for template_widened)
  timestamp: SystemTime,
}

event_type is the field §6.7’s drift-detection query filters on. ourios.miner.merges increments on every event whose event_type is template_widened or template_type_expanded (the two structural widenings); template_widening_rejected_degenerate is recorded but does not increment ourios.miner.merges. The audit stream is written to the same WAL as the data records and ends up in a dedicated audit Parquet file per tenant per compaction window (schema in ourios-parquet’s RFC, not this one).

Default policy: strict. Widening is permitted whenever the clean-attach path in §6.2 would otherwise mismatch positions. The audit event is mandatory — no widening, of any reason, ever proceeds without one. Code paths that would emit a widening without emitting an audit event are blocked at PR review per hazards.md H1.

WAL durability ordering of audit events. A single attach may emit two audit events in order (RFC0001.7: template_widened immediately followed by template_type_expanded) and one data record. The contract this RFC requires from the future ourios-wal RFC is an ordering-plus-durability-barrier: a data record carrying template_version = V must not become durable before every audit event justifying the leaf’s progression to V is durable. Crash recovery may then observe some prefix of [event_1, event_2, …, data_record], but never a state in which the data record exists without the events that caused its version stamp. Any framing strategy that satisfies this — a composite multi-record frame, batched-fsync ordering, a two-phase write-then-link, anything else — is acceptable; the framing is ourios-wal’s choice, the ordering barrier is this RFC’s requirement. Without it, replay would bump template_version fewer times than the in-memory leaf did and the surviving data records would reference a version the audit stream cannot substantiate.

Degenerate template guard. If a widening would leave the template with zero non-wildcard tokens (the entire template becomes <*> <*> … <*>), the widening is rejected, the line is treated as a parse failure (confidence = 0, retain body, increment ourios.miner.parse_failures), and an audit event with event_type = template_widening_rejected_degenerate records the rejection. A fully-wildcard template provides no parsing value and would swallow arbitrary lines.

6.5 Parameter handling [§3.2]

Per-parameter byte limit. Default 256 B, configurable up to 1 KiB (the [§3.2] ceiling). Above 1 KiB requires an RFC.

Overflow behaviour. When a parameter value (post-masking) exceeds the configured limit, the parameter slot is replaced by an OVERFLOW marker:

Param {
  type_tag: ParamType::OVERFLOW,
  value: encode(length: u32, sha256_prefix: [u8; 8]),
}

The original line L_raw is captured into the body column unconditionally (overflow forces body retention, regardless of lossy_flag). The 8-byte SHA-256 prefix lets queries “find rows where this exact long parameter occurred” without storing the long value in the columnar data. Reconstruction honours overflow: reconstruct(record) falls back to body when any param has type_tag == OVERFLOW.

Telemetry. Two metrics for [§3.2] and hazard H2:

• ourios.miner.params.overflow (counter, attributes ourios.tenant, ourios.service): increments per overflow.
• ourios.miner.params.overflow.utilization (gauge, attributes ourios.tenant, ourios.service): rolling overflow rate. Alert at > 0.01 per service per [§3.2].

6.6 Body reconstruction [§3.3]

Amendment 2026-06-08 (reader render contract). H7.3 (§5) referenced “the §6.6 warning marker,” but §6.6 defined no such marker — the prose Reader behaviour paragraph it replaced described one only informally. This amendment adds the Reader render contract subsection below, defining the marker as a structured, out-of-band per-row Reconstruction signal (Faithful | RetainedVerbatim) the reader attaches to the rendered row — never a mutation of the body bytes — and pinning the lossy short-circuit H7.3 requires (return body verbatim, do not call reconstruct). The clean-path read-time template lookup (a registry mapping (template_id, template_version) → tokens at read time) is explicitly out of scope and deferred to the querier’s reader-materialisation story (RFC 0007). RFC 0001 stays specified; this clarifies the §6.6 contract, it does not change the on-disk schema or the mining algorithm.

Capture, always. Every successful tokenization in §6.2 step 1 populates the separators array with the bytes between adjacent tokens (and the leading and trailing bytes of the line). The array length is tokens.len() + 1. There is no whitespace heuristic and no “is this whitespace trivial” decision — the bytes are captured verbatim. Storage cost is bounded (typical separator is one space; the array dictionary-encodes well in Parquet) and the implementation has no fuzzy boundary that could decide to drop bytes silently.

Reconstruction function.

fn reconstruct(record: &Record) -> Bytes {
    if record.lossy_flag {
        return record.body.expect("lossy implies retained body");
    }
    if record.params.iter().any(|p| p.type_tag == OVERFLOW) {
        return record.body.expect("overflow implies retained body");
    }
    let template = lookup(record.template_id, record.template_version);
    let mut out = BytesMut::new();
    out.extend_from_slice(&record.separators[0]);
    for (i, token) in template.tokens.iter().enumerate() {
        match token {
            Token::Fixed(s) => out.extend_from_slice(s),
            Token::Wildcard(slot) => {
                out.extend_from_slice(&record.params[slot].value)
            }
        }
        out.extend_from_slice(&record.separators[i + 1]);
    }
    out.freeze()
}

lossy_flag semantics. Set to true if and only if reconstruction is not guaranteed to equal the ingested bytes:

• The tokenizer failed (malformed UTF-8 inside a token, embedded NUL, line exceeded the configured max_line_bytes cap before tokenization completed).
• A preprocessing rule explicitly rejected the line.

The lossy zone in §6.3 (low confidence) does not automatically set lossy_flag: the body is retained either way, and reconstruction from template + params + separators is still expected to match. The flag is reserved for the cases where the record genuinely cannot be reconstructed.

Reader render contract

The functions above run at write time: reconstruct is the property-test oracle and the in-process renderer the miner exercises while the template is in hand. The reader — the read-side path that materialises a stored Parquet row back into the effective original line for a query result — is a distinct caller, and H7.3 pins the contract it must honour. The contract below covers String-body rows (body_kind = String); structured bodies are out of scope (see the end of this subsection).

For a String-body row, the render result is (bytes, reconstruction). Rendering yields two things: the effective original line bytes and a per-row reconstruction signal

#![allow(unused)]
fn main() {
enum Reconstruction {
    Faithful,         // bytes == ingested line, reconstructed from template
    RetainedVerbatim, // bytes are the retained `body` column, not reconstructed
}
}

The Reconstruction signal is the “§6.6 warning marker” that H7.3 references. It is structured, out-of-band metadata attached to the rendered row — not a mutation of the body bytes. A consumer (the DSL output layer, RFC 0007; a UI) renders RetainedVerbatim as a “rendered from the retained body bytes, not reconstructed” warning beside the row, exactly as it would render any other per-row annotation.

A body-byte annotation is explicitly rejected as the marker. Prefixing the body with a sentinel string, wrapping it in a marker character, or otherwise editing the bytes to carry the warning would break the verbatim guarantee: the whole point of the lossy path is that an operator asking “show me what was actually logged” gets the ingested bytes back unchanged [§3.3]. The marker therefore lives beside the bytes, never inside them.

Lossy path (H7.3). For a row with lossy_flag = true — the tokenizer-failure / explicit-rejection cases enumerated under lossy_flag semantics above — the reader returns the body column verbatim with Reconstruction::RetainedVerbatim, and does not invoke reconstruct: no template lookup, no token walk. The same short-circuit applies to a String-body row carrying an OVERFLOW param (§6.5): its retained body is returned verbatim with Reconstruction::RetainedVerbatim. reconstruct’s own lossy_flag / OVERFLOW early returns remain in place as a defensive guard for callers that reach it anyway; the reader’s contract is to short-circuit before that call, so the guard is belt-and-braces, not the primary mechanism.

Structured bodies are out of scope of this amendment. A body_kind = Structured row renders its body column from the Ourios canonical body encoding (the §6.1 rule, exercised by the RFC0001.9 structured short-circuit). This amendment defines the Reconstruction marker for the implemented String path only; how a structured render maps to a Reconstruction signal is left open here, to be settled when structured-body rendering is wired.

Clean path. For a faithful row (lossy_flag = false, no OVERFLOW param, template available) the reader invokes reconstruct(record) and attaches Reconstruction::Faithful. This path requires resolving (template_id, template_version) → tokens against a template registry available at read time. That registry — how the reader obtains the template a stored row was mined against, given that the miner’s in-memory tree is an ingest-side structure — is a separate concern and out of scope of this amendment. Today reconstruct is exercised only where the template is already in hand (the write-side property test H7.1, and H7.4); the read-time lookup mechanism is future work, tracked with the querier’s reader-materialisation story (RFC 0007). This amendment pins the render contract and the lossy path; it does not specify a template-registry design.

The reader never silently substitutes one rendering for the other: every rendered String-body row carries its Reconstruction signal, and the lossy/clean branch is selected solely by the row’s lossy_flag (and the §6.5 OVERFLOW case), never inferred from the bytes.

Property test. For every row r in the corpus where r.lossy_flag == false:

reconstruct(r) == ingested_bytes(r)

Failure is a build break, not a regression — [§3.3] and hazard H7 both name this as the property test that gates merges.

6.7 Template versioning and drift [§3.5], hazard H5

A template’s structural state changes over time as widenings (§6.4) and parameter-type expansions accrue. Each change increments template_version and emits an audit event of type template_widened or template_type_expanded. ourios.miner.template.version_changes counts these.

Two distinct cross-cutting questions. “Same leaf, different structural snapshots” and “different leaves that mean the same thing” are separate problems, with separate query forms:

• Cross-version (within one leaf). A leaf’s template_id is stable across every widening of that leaf (§6.1, “Template identity”); only template_version advances. So a literal predicate where template_id = X already returns rows from every version of leaf X by construction — no alias resolution required. To pin to a single structural snapshot, query where (template_id, template_version) = (X, V).
• Cross-alias (across leaves). When a deploy changes a log line enough that the miner allocates a new leaf instead of widening the existing one, the operator has two template_ids for what is semantically the same template. Resolving “all rows for the thing X represents” then requires walking an alias set that spans leaves. RFC 0002 §5.4 exposes this as where template_id.resolves_to(X); bare where template_id = X does not follow alias chains.

The data model in §6.1 supports both shapes: cross-version is free because template_id is stable across widenings; cross-alias is served by a separate alias index that maps a representative template_id to the equivalence class of template_ids the operator (or a future inference layer) considers semantically the same template.

Alias index lifecycle. Cross-alias is structurally distinct from cross-version: widening is intra-leaf and increments template_version (one template_id, several versions); aliasing is inter-leaf and groups template_ids the miner allocated separately (different template_ids, the operator asserts they mean the same thing). template_widened events therefore do not populate the alias index — they live on the cross-version axis.

Alias write path: operator-driven and audited [§3.1]. The alias index is operator-driven, never silently inferred. An alias assertion — “leaf B is the same template as leaf A” — is an explicit operator action recorded as an audited, durable event, exactly as §3.1 requires of any merge (“every merge emits an audit event; explicit”). Automatic inference is the precise failure mode §3.1 forbids: an auto-aliased cross-semantic merge is a silent merge by another name. The amendment below replaces the previous “no creation event / produced out of band” deferral with the mechanism.

Amendment (alias write path, 2026-06-07). Resolves the §9 open question “Alias index creation mechanism.” The alias index is produced by operator-driven, audited, reversible assertions on the §6.4 audit stream; automatic inference is deferred to a possible future propose → operator-confirm layer (proposals never enter the active index unconfirmed). Adds the alias_asserted / alias_retracted audit events, the per-tenant alias-map projection the querier reads, and §5 scenarios RFC0001.12–RFC0001.16. The RFC re-enters the ladder at specified until those scenarios land (precedent: the RFC 0003 served-binary amendment).

The alias model. An alias set is an equivalence class of template_ids within one tenant [§3.7]. Membership — which template_ids are in the class — is the only thing that carries contract weight: resolves_to expands by membership and nothing else (RFC0001.13). The class also has a canonical representative, defined as the numerically smallest member of the materialized set. The canonical representative is a derived display/identity convenience, not what defines membership: it gives the set a stable name and is deterministic regardless of assertion order, but it plays no part in deciding who belongs to the class. This derivation rule is an evolvable implementation detail, not a contract — it may change (for example, letting an operator designate a preferred representative) without changing alias-set semantics, precisely because resolves_to expands by membership regardless of which member is canonical. (This is distinct from the event-level representative_id below, which is merely the operator’s anchor id for an assertion and need not equal the derived canonical.) Membership is cross-leaf only: it groups template_ids the miner allocated as separate leaves. It never crosses the cross-version axis — every version of a single leaf already shares one template_id (§6.1, “Template identity”), so template_version is not an alias concern and the alias index holds no template_version field. The two axes stay disjoint: widenings move a leaf along template_version; aliasing groups distinct template_ids.

The assertion event. Aliasing is expressed by two new AuditEventType variants on the §6.4 audit stream, carrying an alias-specific payload: