Chapter 08

Sidecar / OTel integration

Reflects Holonograph v0.26.0 · @holonograph/client 0.3.0 · ~9 min read

The lens sees every call that goes through it. Real systems have work that doesn't go through it: a job on a queue, a background worker, a webhook fanning out to three services. This chapter is about carrying the two things the lens needs (run mode, correlation id) across those boundaries via OpenTelemetry — and, when the call path gets hot enough that synchronous delivery starts to matter, how to export evaluation events as OTLP spans with counted-loss delivery-completeness accounting.

What it is

Holonograph composes on top of OpenTelemetry rather than replacing it. Two pieces of the OTel spec do the work:

• Baggage carries key-value context alongside a trace. Holonograph reads and writes two Baggage keys (holonograph.runmode, holonograph.correlation-id) so the run mode and correlation id survive async boundaries automatically.
• Auto-instrumentation covers your framework's normal request/response cycle, so HTTP calls, queue publishes, and pub-sub messages carry the OTel context without you writing propagation code by hand.

Where auto-instrumentation doesn't reach, there's a manual fallback: encode the same values as message attributes on the payload and reconstruct them on the receiving side.

Why it matters

The four-source attribution downstream only works if the events on both sides of an async hop can be tied back to the same call. If the queue drops your run mode, the worker's call defaults to whatever the worker's client was constructed with, often the wrong thing. Baggage keeps that from happening quietly.

How to use it

Enable OTel in your service

Use any OTel SDK for your language. Enable auto-instrumentation for the transports you use (HTTP, gRPC, your queue library, your pub-sub client). Nothing Holonograph-specific here; the standard OTel setup is enough.

Set Baggage on the originating side

At the top of the request that started the work, put the run mode and correlation id into Baggage:

import { propagation, context } from "@opentelemetry/api";

const baggage = propagation.getBaggage(context.active())
  ?? propagation.createBaggage();

const withHolonograph = baggage
  .setEntry("holonograph.runmode",       { value: "production" })
  .setEntry("holonograph.correlation-id",{ value: correlationId });

const ctx = propagation.setBaggage(context.active(), withHolonograph);

// run the rest of the request under `ctx`
context.with(ctx, async () => {
  await enqueueBillingReview({ userId });
});

Auto-instrumentation on the queue publish serializes Baggage into the message; no message-body changes required.

Read Baggage on the receiving side

The worker or subscriber picks up the Baggage after auto-instrumentation extracts it. Use it to construct the HolonographClient:

import { propagation, context } from "@opentelemetry/api";

async function handleJob(job) {
  const baggage = propagation.getBaggage(context.active());
  const runMode = baggage?.getEntry("holonograph.runmode")?.value ?? "local_dev";
  const correlationId = baggage?.getEntry("holonograph.correlation-id")?.value;

  const client = new HolonographClient({
    endpoint: "http://127.0.0.1:8080",
    runMode,
    lensVersion: "billing-lens/v14",
    substrate: { /* ... */ }
  });

  const handle = await client.messages.create({
    surfaceId: "review-flagged-charge",
    messages: [ /* ... */ ],
    correlationId
  });
  // ...
}

The call in the worker now shares a correlationId with the request that produced the job, and it inherits the originator's run mode, so a test request that enqueued a job doesn't turn into a production call on the worker.

Message-attribute fallback

For transports where Baggage propagation isn't automatic (some managed queues, some pub-sub platforms), encode the two values as explicit message attributes on the payload, and reconstruct them on the receiving side before setting Baggage manually.

await queue.publish({
  body: { userId },
  attributes: {
    "holonograph.runmode": "production",
    "holonograph.correlation-id": correlationId
  }
});

On the receiver, read the attributes and set them into Baggage the same way you would in a fully-instrumented setup. The rest of your code doesn't need to know which path they took.

Reading the output

Nothing changes at read time. Events written from the worker come back through GET /lens/events with the same correlationId as the originating request, and their run_mode column matches the originator's. That's what makes them queryable as one evaluation across the async boundary.

Bounded emit buffer and OTLP export

Baggage carries the run mode and correlation id across async hops. Delivery of the evaluation event itself is a separate concern. The @holonograph/client 0.3.0 release adds a bounded emit buffer with three drain modes — direct HTTP, OpenTelemetry OTLP spans, or both — so operators can move evaluation events off the request path when synchronous POST /lens/events is too expensive to await.

The buffer is bounded, drops the oldest events when full, and counts every drop. Delivery completeness is a first-class read: coverage is a floor, not an assumption. See the SDK reference for the full emit option shape; this section is about the wire.

The four delivery modes

sync is the default. The three buffered modes share the same ring buffer, the same accounting window, and the same drop semantics; only the drain differs.

What lands on the OTLP wire

Under otlp and dual, each evaluation becomes one span. Its dimensions become span events. Every attribute is namespaced under holonograph.*, so nothing collides with the rest of your OTel signal.

Attributes on the evaluation span

Per-dimension span events

Each graded dimension is a span event named holonograph.dimension with:

Session sentinels: gap detection

Every emit.sentinelIntervalMs (default 30s), the client emits a holonograph.session.sentinel span on span-carrying channels. Sentinels carry emission.stream and emission.seq just like evaluation spans. Their purpose is gap detection: a consumer that sees an ordinal jump knows an event was dropped, even if the client crashed before it could tell anyone. The emission.final attribute goes true on the last event emitted by a graceful shutdown(), so a consumer can distinguish "the client stopped cleanly" from "the connection went dead."

Delivery completeness: the read

A bounded buffer with drop-on-overflow only produces trustworthy analysis if the drops are visible. The client publishes an accounting window every emit.windowMs (default 10s), and exposes it two ways:

• Pull: client.completenessWindow() returns the current window snapshot.
• Push: emit.onWindow(window) fires once per completed window.

The window carries offered / dropped / delivered / failed counters, a per-channel breakdown (so http loss and otlp loss are distinguishable), a completenessLowerBound ratio, and a biasFlag that goes true whenever drops occurred. Downstream analyses read biasFlag and treat the affected interval as potentially non-random. See SDK reference § Delivery modes for the full CompletenessWindow shape.

Extreme sustained load

The buffer is designed for occasional overflow, not sustained overrun. When the drop rate exceeds emit.extremeSustainedLoad.dropRateThreshold (default 1%) for longer than emit.extremeSustainedLoad.sustainedForMs (default 60s), the client fires onExtremeSustainedLoad with a summary event. Route it to your alerting; that is the signal to raise the bufferSize, add drain workers, or step down evaluation coverage rather than let the sample bias grow silently.

Graceful shutdown

Under buffered modes, call client.shutdown() before the process exits. It flushes any remaining buffered events, stops the drain, and marks the last-emitted span as emission.final = true so downstream consumers know the stream ended cleanly. No-op under sync.

process.on("SIGTERM", async () => {
  await client.shutdown();
  process.exit(0);
});

Reference

Baggage keys

Message-attribute keys

Same keys as above, used verbatim as message attributes when Baggage propagation isn't available on the transport.