Agentic behavioral attribution

Your agent, the model, evals, or noise?

Holonograph attributes agentic behavior to the underlying source.

Runs Local·Statistical Algorithms·Honest Attribution

One behavior tracked across every version

The longitudinal substrate

Attribution requires memory.

A number can move without the behavior moving. When the model is updated, or the lens that scores it is revised, the metric drifts while the underlying behavior does not change. Holonograph versions every lens and keeps the full history, so each reading is checked against the exact lens that produced it. The outcome is a clear behavioral attribution to the model, the substrate, or the judges themselves.

The method

Identifying the source of the shift.

A value changing is not yet an answer. Holonograph decomposes a score by attributing each part to your agent, the model, or custom evaluation system, and separates real change from noise; quantifying how much each source moved with that number. This decomposition is the method under patent application. It turns the changed score into an actionable report. Substrate labeling further identifies which component of your agent’s substrate contributed to the agent’s part: lessons, skills, code, and prompt are each separated and attributed.

An observed shift

The instrument

your agent

The prompts, tools, and logic you built.

the model

The provider’s weights, version, and token relationships. Aspects you can’t control.

your eval

The judge and/or rubric doing the scoring. If a judge, also subject to the same model black box as above.

noise

Sampling and chance, identified and isolated.

Past a certain scale, a metric that moves is not an answer. Attribution is.

Facets of rubric, judge, and context scope

Lens facets

A rubric doing too much stops discriminating.

When one judge scores many concerns at once, its scale can come to rest at an anchor. A perfect ten and a nine stop being different readings, and a failure no longer tells you how many things went wrong, only that something did. Holonograph reports that condition rather than passing the number through as an ordinary score. Where a rubric resolves into separate facets of rubric, judge, and context scope, each is grouped and attributed on its own, and where two move together the apparatus says so instead of guessing between them.

The science of lens facets →

The analysis

Every finding names its source.

The instrument returns the effect, the statistics behind it, and an honest account of what it cannot observe or separate, and clear instructions on how to achieve that state of observability.

Example readings*
Needs review Hidden factor life.chat-response-v1 · helpfulness ×240 · 10d Untriaged

Helpfulness changes by 4.3 points depending on region.

region was declared an axis that shouldn’t move it at all.orthogonality

Average helpfulness score, by region · scale 0–10 · overall average 6.25

us8.40n=120
eu4.10n=120

✓240 turns across 2 groups. A gap this wide would show up by chance less than once in 1,000 samples.

This can’t tell an agent problem from an eval problem.

The agent really is worse for some region values.

The judge is rewarding something that travels with region.

Separating the two takes its own probe.

effect size31%share of the helpfulness swing traced to region
bias-corrected0.28holds after correcting for small samples, so it isn’t a group-size artifact
adjusted p<0.001after penalising every other check run on this surface
scored240 · 2 grpturns carrying a helpfulness score and a region value
gatedependence-robustmultiple-comparison correction, re-derived on every read
Attributed Citation check life.chat-response-v1 · faithfulness ×180 · 10d Root cause: agent

One in six answers cites a source the agent never retrieved.

The agent was told to call hn_search before citing. On 31 turns it answered from memory.tool-discipline

Answers citing a source, by whether the trace backs it · 180 total

backed14983%
unbacked3117%

✓The trace shows no hn_search call on those 31 turns. This is a record of what ran, not an estimate.

This one it can separate. The source is the agent.

Not the model: it returned a valid answer.

Not your eval: the judge scored what it was shown.

The agent skipped the retrieval it was told to run.

effect size17%share of citing answers the trace can’t back
attributedagentsource resolved from the trace, not inferred from the score
coverage180 / 180every citing turn checked against its own trace
window10 dayscontinuous, the failure rate held across it
gatetrace-linkedevery turn opens to the full run that produced it
The effect

What moved, and by how much.

The rule it broke

You declared region can’t move this score.

Not chance

A gap this wide is one in a thousand.

Honest by design

No guessing. It names the next probe.

Auditable

Every statistic, re-derived on read.

The effect

The agent fabricated a citation.

The rule it broke

Cite only what you retrieved.

Caught by the trace

The retrieval tool was never called.

Attributed

The source resolves to the agent.

Auditable

Every turn opens to its trace.

* The example readings above are illustrative, showing the format of a reading in the instrument’s own layout rather than measurements from a specific deployment.

See it live

The instrument, running.

This is the Act queue against a live agent: every finding it surfaces, the moment it fires. Nigel will show you around.

LIVE

The live readout runs best on desktop.

Open Holonograph on a larger screen to watch findings fire in real time. Nigel can still answer your questions here.

Ask Nigel →