Chapter 01
Overview & first principles
Holonograph is the observation layer for agentic AI systems. It sits in the call path between your agent and the language models that drive it, and it captures a structured record of every evaluation your system produces. From that record it tells you two things existing tools cannot: which layer of your system a change came from, and how much of an observed spread is real change versus measurement noise.
What it is
A signed, notarized native binary that runs as a localhost daemon inside your own infrastructure. It ships as a bidirectional gateway at the wire-format boundary between your agent and the model — every request out and every response back passes through the lens.
Because it owns that boundary, Holonograph can do things a library bolted onto your agent structurally cannot:
- Record the whole call, in a shape that stays consistent across vendors and model versions.
- Multiplex — run one call against several vendors at once and compare them head-to-head on speed, price, and accuracy — while the agent sees only the primary's reply.
- Attribute a change to the layer that produced it: your deployment, the vendor's model, your evaluation configuration, or the model's irreducible noise.
- Isolate variance — separate real movement from measurement noise, so you know whether a change is worth acting on.
- Close the loop — captured overrides and approvals cluster into drafted corrective artifacts (skills, lessons, fixtures) that ship through a human-approval gate.
Every one of those is a capability the operator uses. This Guide teaches you how to use them.
Why position, not instrumentation
Observability tools for LLM systems have converged on a pattern: sprinkle logging calls through the agent's code, ship traces to a dashboard. That pattern works when the thing you're observing is deterministic. Modern agentic systems built on foundation LLMs are not — the model can be silently re-routed by the vendor behind a stable alias, and the evaluation apparatus is itself an LLM that drifts on its own schedule. Instrumentation inside the agent's code can only see what the agent chose to log, and the log format ages the day the SDK changes.
Holonograph takes a different premise: observation is a property of architectural position. Because the lens sits at the wire boundary, no logging call is required, no SDK version is coupled to the record, and nothing the agent does at the model boundary is structurally invisible. An OpenTelemetry sidecar covers anything that doesn't pass through the lens (async work, message queues), so the practical guarantee is observational completeness: every call is either graded by the lens or captured by the sidecar.
What comes back
Holonograph exposes four operator-facing surfaces. Each is documented in its own chapter, but knowing they exist is what makes the rest of the Guide make sense.
1. The four-layer snapshot
The atomic unit Holonograph writes for every evaluation. One record binds the layers of that evaluation — the surface that was called, the model that answered, the outcome you reported, the state of your deployment — into a shape you can query as history.
2. Drift attribution
When a score changes over time, drift attribution names one of four sources as the likely cause — substrate (your own deployment), light source (the vendor moved under you), lens (your evaluation configuration changed), or noise (the model's irreducible non-determinism). What you do about a change depends on which label it wears; that's the Drift attribution chapter's job to walk through.
3. Variance isolation
When your evaluation scores move, some of the movement is real change in your system and some is just the noise of measuring with a language model. Variance isolation returns a cleaned reading and a trust signal that tells you whether to act on it. The specifics — how to read it, how to hold one still — are the Variance isolation chapter's job.
4. The lessons pipeline
Captured overrides (a human edited a draft) and approvals (a human sent it as-is) cluster together. The system drafts a concrete corrective artifact for each cluster — a new skill, lesson, fixture, or code-change recommendation — and nothing ships without a named approver at the gate. Once published, the artifact becomes substrate: versioned, captured, and attributable in future drift readings like any other change.
What Holonograph is not
Holonograph is frequently confused with adjacent tools. It is none of these:
- Not concept-drift detection — that field assumes a deterministic model and a measurable shift in input data. Holonograph addresses non-deterministic foundation LLMs where the model itself can be silently re-routed by the vendor.
- Not feature attribution — SHAP, LIME, and integrated gradients attribute a single output to its input features. Holonograph attributes pass-rate changes over time to one of four sources of change.
- Not an MLOps dashboard — not a tracing system, not a model registry, not a metrics platform. Holonograph composes on top of OpenTelemetry; the novel layer is the versioned lens and the attribution it enables.
- Not LLM-as-judge — conventional LLM-as-judge treats the judge as a fixed measuring stick. Holonograph treats the judge itself as a first-class versioned instrument whose drift is independently attributable.
- Not a fine-tuning pipeline — the lessons pipeline produces operator-curated artifacts that ship through a human-approval gate. It does not adjust model weights.
- Not SaaS — a signed, notarized native binary that runs as a localhost daemon. Data never leaves your infrastructure.
How to read this Guide
One spine of pages, four depths. Every chapter follows the same internal shape — What it is, Why it matters, How to use it, Reading the output, Reference — so a CTO can read only the first two sections of a chapter and leave informed, while a developer working from the same page can jump straight to How to use it and go.
The audience router on the index page points each role at its entry point. If in doubt, follow the developer path — every other role can skim from there.
Where the wall is
The Guide teaches you to operate the instrument. It does not teach you to rebuild it. The line already exists in the software: the SDK exposes a contract (call this, get that back) and hides the mechanism behind it. This Guide documents the contract.
Concretely: the underlying method is deliberately absent from these pages. If you are a researcher trying to evaluate the method rather than the product, the shortest path is to talk to us — deeper technical validation is available under NDA, and the product itself is designed to run in your own environment against your own workload, so nothing about the evaluation happens on our infrastructure.