Chapter 22
Compaction & husking
Compaction removes the bulky payload from an aging event and keeps every structured field any estimator reads. The lens never compacts on its own.
What it is
An operator-invoked pipeline: plan, verify, seal, move. Three endpoints, and the first of them writes nothing.
| Endpoint | Behavior |
|---|---|
POST /lens/retention/plan | Always a dry run. Reports what would be compacted and writes nothing. |
POST /lens/retention/compact | Executes an approved plan. Approver-gated. |
POST /lens/retention/unseal | Undoes an archive-mode seal. |
Retention classes
Every event kind declares one. The enumeration is exhaustive, so a new event kind fails the type check until it has been classified.
| Class | Meaning |
|---|---|
never | Never compacted. Evidence, markers and seals. |
husk | Payload may be archived or deleted; structured fields are kept. |
summarise | Compactable only into lossless summaries. |
Why it matters
A husk is a projection, not a new fact
Husking keeps every structured field an estimator reads: ids, occurredAt, the whole
substrate, the surface grading including dimension scores, judge identity and structured claims,
gate outcomes, cohort tags, light identity and accounting. What leaves is bulk:
light.rawRequest, rawResponse, reasoning,
toolCalls, vendorMetadata, error.message, each dimension's
rationale, sideEffects[].details, and oversized unknown keys.
Where the payload went is recorded on the husk. archived means a sealed,
hash-chained archive segment holds the full event and the operator can unseal it.
deleted means it is gone and only the digest remains. erased means a
right-to-delete request removed it.
The stamp cannot be forged
A husk stamp is server-projected. It exists only on husks the compactor wrote into a sealed husk segment, it is never written onto a stored raw event, and every producer ingest path rejects an event that arrives carrying one. A forged stamp would claim facts the lens never extracted.
Payload-reading consumers keep working
Some consumers read the payload: verdict reliability, co-groundedness, trajectory, exemplars. The
husk carries a facts block holding those extractors' outputs, computed over the raw
event at seal time and stamped with each extractor's version. A consumer whose extractor version
or configuration no longer matches reports a method boundary rather than a clean result.
How to use it
- Plan first and read the plan. It is a dry run by construction, not by flag.
- Compact against an approved plan. The approval is the gate.
- Prefer archive mode where the payload may be wanted again; an archive seal can be unsealed, a delete cannot.
- Every consumer of event history declares its post-compaction fidelity, so check that declaration before compacting a span an analysis still depends on.
Reading the output
A compacted span reads the same as an uncompacted one for every structured consumer. Scores, grading, gate outcomes and identity are unchanged, because none of them lived in the payload. A reading that cannot be reproduced on a husk says so as a method boundary.
Reference
- On Firestore, compaction is opt-in and never edits a record in order to build its index.
- The judge-verdict linking model is husk-invariant: a husked event resolves the same verdict a whole one does.
- Retention policy and schedule are stored durably and are operator-owned.