An SA's honest take Field notes from live delivery
01 Architecture Aug 20, 2026 6 min read

Stop climbing the logging ladder. The top is already built.

OTel tracing is what structured logging was always trying to become. The top rung ships as a product — what keeps teams three rungs down is comprehension and prioritisation, not engineering effort.

The observability argument is usually fought as instrumentation — which SDK, which agent, how much application code you have to touch — when instrumentation stopped being the problem years ago. OTel won. Everything ingests OTLP now. What I keep walking into is teams stalled three or four rungs below that, and not for any technical reason.

The ladder is a description, not a plan

Almost every logging setup I see sits somewhere on this:

  1. A log message.
  2. Add a level and a timestamp.
  3. Add key=value pairs so you can grep them out.
  4. Move to JSON so you can stop grepping.
  5. Add request context — tenant, route, build.
  6. Add a duration: a start and an end.
  7. Add a parent, so nested work points at its caller.
  8. Standardise the field names and propagate the IDs across the network.

Step 8 is OpenTelemetry. That’s the whole trick.

A span is a structured log line with a duration and a parent pointer. Read the OTLP schema and it’s sitting in the open: trace_id, span_id, parent_span_id, a start and end timestamp, and a bag of typed attributes. OTel’s own log records carry trace_id and span_id as well. The two data models were deliberately collapsed into one — the spec is telling you they’re the same thing.

The one genuinely hard rung is 7→8, and it isn’t formatting. It’s context propagation: carrying traceparent across every process, queue and library you don’t own, plus ambient context so you aren’t threading IDs through every function signature. That’s a protocol, not a logger config.

The hardest step on the ladder is the one someone else already finished. The SDKs, the collector, Refinery, the backend — the top rung is a dependency, not a project.

Nobody is stuck for engineering reasons

Two misreadings keep teams down the ladder, and both are conceptual.

The first is the word distributed. Call it distributed tracing and every team running one deployable concludes it isn’t for them. But your monolith is already a distributed system — it just isn’t distributed at the layer you write code in. It talks to a database, publishes to a queue, calls two SaaS APIs, and runs in a pod on a node in a cluster. Auto-instrumentation gives you every one of those hops without touching application code: database client spans with the query attached, producer and consumer spans with context propagated through message headers, and pod, node and namespace stamped on by the collector. The first trace a monolith team looks at usually contains an N+1 — the same query, repeated in a long identical row, obvious at a glance. No amount of log aggregation ever made that visible.

The second is thinking a span is a special artifact you emit for performance work. It isn’t. It’s your log line, with context attached and a duration on it. If you can write a log statement, you already know how to instrument. That single reframe is worth more than any workshop on trace semantics.

Then there’s the part nobody puts on a slide: this is process effort, not engineering effort. Dropping in a new format and migrating is boring, mechanical, and finite. What’s hard is that evolving logging has no ticket, no owner and no quarter — it never wins against a feature. And in a large, geographically distributed enterprise it gets worse, because someone reasonably proposes agreeing a house standard first, and that becomes a governance exercise spanning four time zones and outliving two reorgs. The joke is that the standard already exists. OTel’s semantic conventions are the output of exactly that argument, held in public, by people doing it full time. You can adopt the conclusion without running the meeting.

The part the vendor list doesn’t tell you

Jumping to rung eight only pays off if the backend treats what arrives as a record rather than a picture of one. “Supports OpenTelemetry” is a parser. It says nothing about the storage model underneath, which is what determines whether you can answer a question you didn’t anticipate.

Fig. 1 — Storage model // what it decides
 Trace viewerEvent store
Finding a traceYou need the IDYou query for it
Group by an attribute you never declaredNoYes
High-cardinality dimension (user, tenant, build)Cost, or refusalIt’s just a column
Logs, metrics and tracesThree stores, three billsOne

Most products land on the left for two unglamorous reasons. Sunk architecture: an inverted index, a TSDB and blob-by-trace-ID can’t be unified by shipping a new dashboard — Dynatrace needed a full re-platform onto Grail to get there. And the revenue model: separate SKUs for log ingest, APM hosts and custom metrics only make sense if the data lives apart. Honeycomb is the clearest example of the right-hand column, largely because it started there and had no other stores to protect. Jaeger and Tempo sit firmly on the left, deliberately — they’re trace viewers, and good ones. Different product, not a worse one.

What I’d check before signing:

  • Can you group by an attribute nobody declared in advance? If adding a dimension needs a schema change or a new custom metric SKU, you bought a metrics product with a waterfall view.
  • Does the backend weight sampled spans? Tail sampling that keeps every error and 1% of successes produces a nonsense error rate unless the store multiplies by the recorded sample rate. Invisible and load-bearing.
  • Aggregate before you sample. The collector’s spanmetrics connector derives exact RED metrics from 100% of spans, then you sample the spans themselves. Exact counts and representative traces — you don’t have to choose.
  • Keep real metrics for real infrastructure. CPU, memory, GC and disk have no request context to hang off. Everything request-scoped should be an aggregation over span attributes instead.

Nobody needs to be sold on tracing anymore. But the blockers were never in the code — they’re in the heads. What people think tracing is for, whether they believe it applies to one application, and whether anyone will own a change nobody gets paged about. The implementation is the simple part. It always was.