What Is Observability - and When Plain Monitoring Is Already Enough
Ask five engineers what is observability and you will get three vendor pitches and two definitions. The useful definition: observability is the property of a system that lets you answer new questions about its behaviour using the data it already emits, without shipping new code to go and look.
That phrasing matters because it names the problem. In a system with one service and one database, you know the failure modes in advance, so you can build a dashboard for each one. In a system with a dozen services, async hops and daily deploys, the failure space is combinatorial — the question that resolves the next incident is one nobody thought to pre-compute.
So the design goal shifts. Instead of pre-aggregating a fixed set of answers, you keep enough context per unit of work that an arbitrary question can be asked later. That is the whole idea, and everything else on this page is mechanism or cost.
| Concept | Observability (o11y) |
|---|---|
| Where it sits | Cross-cutting operational concern - lives in your application code first, in a collection and query layer second |
| In one sentence | Observability is how well you can explain a system's behaviour from the data it already emits, without having to add new instrumentation and wait for the problem to happen again. |
The 60-Second Answer
What it does: keeps enough per-request context (structured events, trace IDs, high-cardinality fields) that you can slice production behaviour by dimensions you did not choose in advance.
The one problem it solves: incidents where you know something is slow or failing but cannot attribute it — which tenant, which deploy, which dependency, which code path.
The one cost it adds: data volume you pay for twice — storage/egress, and query time. Controlling it means sampling and retention decisions, which are permanent operational work, not a one-off setup.
Decide by: if your last three incidents were diagnosed by adding logging and waiting for a repeat, you need it. If they were diagnosed by reading one log file, you do not.
How It Actually Works
The mechanism, step by step
- 1. Nothing is observable unless the code emits it
- 2. One wide structured event per unit of work
- 3. High cardinality is the point, not a side effect
- 4. Context propagation stitches one request across services
- 5. Sampling is how you afford any of this
- 6. Aggregation moves to read time
- 7. The loop closes: alert on aggregates, explain with events
1. Nothing is observable unless the code emits it
Instrumentation comes before tooling
Observability is not something you point at a running system from the outside. It is a property the system has because its code emits the right facts. Library-level auto-instrumentation gets you the skeleton for free — inbound HTTP handlers, database driver calls, queue consumers — which tells you the shape of a request but almost nothing about your domain.
The fields that actually explain an incident are domain fields: tenant, plan tier, feature flag state, retry count, shard, cache hit or miss. Those exist only because someone added them.
⚠️ In production — Auto-instrumentation makes week one look finished. The first real incident is where you learn that every span is labelled POST /api/v1/orders and not one of them says which customer or which branch of the code ran.
2. One wide structured event per unit of work
Events with fields, not sentences
Rather than scattering log lines through a request, emit a single structured event when the unit of work ends, carrying everything known by then. Every field is a dimension you can filter and group by later.
{"event":"order.create","status":500,"duration_ms":812,"tenant":"acme","plan":"free","region":"eu-west","db_pool_wait_ms":640,"retries":2,"cache":"miss","build":"a91f3c"}
One row like this answers more questions than a page of prose logs, because the prose has to be parsed and the row does not.
⚠️ In production — The moment a value is baked into a formatted string — "request took 812ms for acme" — it stops being queryable, and someone will write a fragile regex to get it back. The reverse hazard: once a field exists, dashboards depend on it, and renaming it breaks them silently.
3. High cardinality is the point, not a side effect
Unique IDs are the useful dimensions
Cardinality is the number of distinct values a field can take. region is low, user_id and build are high. The questions that end incidents are nearly always cardinality-bound: which tenant, which deploy, which shard.
This is the sharpest technical split in observability vs monitoring. A time-series metrics store creates a separate series per label combination, so high-cardinality labels multiply its memory and index cost. An event store is built to hold those values as ordinary columns.
⚠️ In production — Putting user_id on a metric label is the classic way to take down your own metrics backend — series count grows with your user base and does not shrink when traffic does. Identifiers belong in events and spans; metric labels stay bounded and enumerable.
4. Context propagation stitches one request across services
A trace ID rides along the call
A single service's events are not enough once work crosses a process boundary. Each service creates a span, and passes the trace ID plus its own span ID to the next hop in a header; the callee makes its span a child of that one. The resulting tree is a trace, and it is what gives you causal ordering rather than a pile of timestamps.
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
⚠️ In production — Propagation breaks at every boundary your instrumentation does not own: a queue whose message envelope drops unknown headers, a thread pool or background task that loses the context-local, a proxy or third-party webhook in the middle. It fails silently — you get two unrelated half-traces and no error telling you they belong together.
5. Sampling is how you afford any of this
Keep the interesting traces, drop the boring ones
At real traffic volume, retaining every trace costs more than running the system that produces them. Head sampling decides at the start of the request — cheap, but it throws away errors it could not yet see. Tail sampling buffers the finished trace and then decides, so you can keep all errors and slow outliers plus a fraction of healthy traffic.
The tradeoff shape: head sampling costs you fidelity on rare events, tail sampling costs you buffering memory and a collection tier that has to hold whole traces in flight.
⚠️ In production — Sampling turns counts into estimates. Computing an error rate from sampled traces without weighting by sample rate gives a confidently wrong number. Derive rates from metrics, and use traces only to explain them.
6. Aggregation moves to read time
Store raw, slice on the way out
Monitoring pre-computes: you choose the dimensions up front, roll the data up, and store a compact series. Observability keeps events close to raw and groups them at query time, which is exactly what lets you ask a question you had not anticipated.
That choice relocates the cost. The write path gets simpler and cheaper to reason about; storage volume and the query engine get expensive, and retention becomes a budget decision rather than a technical one.
⚠️ In production — "Just one more group by over 30 days" is what actually eats the bill, and it gets slow long before it gets blocked. Most mature setups run both layers: bounded metrics for the dashboards read every day, wide events for the one hour a week somebody is debugging.
7. The loop closes: alert on aggregates, explain with events
Detect cheaply, diagnose widely
The two layers have different jobs. An SLO on error ratio or latency, computed from cheap aggregates, tells you something is wrong with bounded and predictable cost. From that alert you pivot into the raw events or exemplar traces behind the bad window to find out what.
If that pivot is not close to one click — same IDs, same time range, same tool chain — nobody performs it at 3am, and the expensive half of your pipeline goes unused.
⚠️ In production — Teams that alert on causes (CPU, pod restarts, queue depth) instead of symptoms (error ratio, latency against an SLO) end up with a wall of dashboards built for last year's outage. And an alert with no path into raw events is just a pager telling a human to go and look manually.
## Where You Meet It
- A checkout flow's p99 latency doubles an hour after a deploy, but the service-level dashboards look normal because the median is fine. Grouping the request events by
buildandpayment_providershows it is one provider's calls on the new build, retrying twice before succeeding. Neither dimension was on any pre-built dashboard — that is the part observability bought. - A multi-tenant SaaS gets scattered 'the app is slow' tickets while aggregate latency is flat. Breaking latency down by
tenantshows one customer's bulk import holding connections from a shared pool, so everyone sharing that pool is paying for it. The fix is a pool or rate limit per tenant, but you cannot even see the noisy neighbour without the tenant dimension on every request. - An order moves from an HTTP handler into a queue, gets picked up by a worker, and never produces a result. Because the trace context was propagated in the message envelope, the trace shows the producer span completing and the consumer span starting and being retried on a dead-letter path — rather than two services each claiming innocence.
- A mobile client throws errors only on one app version behind one mobile carrier's proxy. Server logs show TLS resets with no pattern until the client-side events carrying
app_versionandnetwork_typeare correlated with them. These are high-cardinality client dimensions that a metrics-only setup would have had to drop. - An engineer three days into the job is on call. Instead of relying on tribal knowledge of which of eleven services touches a signup, they open one trace and read the actual call path. Observability working as documentation is an underrated part of the payoff.
When You Actually Need It
- Your last few incidents were diagnosed by adding log lines, deploying, and waiting for the problem to happen again. That cycle is the signal that the data you keep cannot answer new questions.
- A user request crosses three or more services or an async hop, so end-to-end latency cannot be attributed to any single process from its own logs.
- You cannot answer 'which customers were affected, and for how long' from existing tooling without hand-querying the production database.
- You are multi-tenant or have per-customer commitments, so a blended average actively hides the pain of the accounts that matter most.
- You deploy multiple times a day, which makes 'which build' a dimension you need on everything and makes pre-built per-failure dashboards obsolete as fast as you write them.
- Each postmortem's action item is a new dashboard, and nobody has looked at the previous ones since the week they were created.
When You Don't
- One deployable, one datastore, one team. If an incident is resolved by reading a single structured log file, you do not need a trace pipeline. Use structured request logs you can grep, an uptime check, and the four golden signals (traffic, errors, latency, saturation) as metrics. Below roughly one deployable and one shared database, distributed tracing adds a system to operate and answers questions you were not struggling with.
- You are missing the cheap basics. No uptime check, no error tracker with stack traces, no deploy markers on your graphs? Buy those first. An error tracker costs a fraction of an observability platform and resolves the majority of application bugs, because the stack trace is the answer. Tracing is for problems where no single stack trace explains the behaviour.
- Bugs you can reproduce locally. If the failure reproduces on a laptop, a debugger beats a trace every time. Spend the effort on a realistic local or staging environment instead of on production telemetry.
- Cron jobs, batch pipelines, and internal tools with nobody waiting. A job-level event row (started, finished, rows processed, exit status) plus an alert on 'did not finish by the expected time' covers nearly all of it. Per-span tracing inside a nightly batch job is effort with no on-call payoff.
- Nobody will query it. With no on-call rotation and no one fluent in the query language, a platform becomes a monthly bill and a dashboard screenshot. Start smaller: add
tenant,user_idandbuildto the structured logs you already have, and see whether anyone uses them. - The budget forces thin coverage everywhere. If affording a platform means aggressive head sampling and a two-day retention window, you will have neither trustworthy aggregates nor the specific outlier you needed. Full-fidelity wide events on your few critical endpoints beat 2% sampling across everything.
What People Get Wrong
"We have observability - we bought a platform for it."
Observability is a property of your system, not a product in your stack. A platform ingesting spans whose only distinguishing field is the route name is an expensive log viewer. The platform supplies storage and a query engine; the useful dimensions have to come from your code.
"There are three pillars - logs, metrics and traces. Collect all three and you are done."
'Three pillars' is a packaging story, and it is mostly how telemetry gets sold. What determines whether you can answer a new question is correlation: a shared trace ID and consistent resource attributes across all three. Three signals in three tools that cannot be joined is three bills, not observability.
"Observability vs monitoring means monitoring is obsolete."
They answer different classes of question. Monitoring answers known questions at bounded, predictable cost, which is exactly what you want for alerting, capacity planning and SLOs. Observability answers unknown questions at variable cost. Nearly every serious setup runs both, and alerting on raw events instead of aggregates is how you get a slow, expensive pager.
"More telemetry means more observability."
Volume without structure just raises cost and query latency. One event per request with twenty meaningful fields is worth more than gigabytes of unstructured text, because you can group by those fields. The thing being maximised is the number of distinct questions answerable, not bytes ingested.
Why It Was Built This Way
Traditional monitoring was designed under two constraints that no longer hold. Storage and indexing were expensive relative to compute, so you aggregated on the write path and kept a small time series. And systems were a handful of long-lived machines whose failure modes were enumerable — disk full, process dead, CPU pegged — so choosing dimensions in advance was a reasonable bet. Under those constraints, pre-aggregation is not a limitation, it is good engineering: a counter per host per minute answers the questions that exist, forever, for almost nothing.
What broke the bet was not scale in bytes but the growth of the question space. Once a request fans out across many services, each with its own deploy cadence, dependencies and tenants, the number of plausible explanations for 'some requests are slow' grows combinatorially. You cannot pre-aggregate along a dimension you have not thought of, and the dimension you need is almost always the one that only became relevant this morning. So the design inverted: keep the unit of work intact with as much context as you can attach, and defer aggregation to read time.
What was traded away is worth stating plainly, because the vendor framing hides it. You gave up bounded cost — query-time aggregation over raw events means your bill and your query latency scale with retention and cardinality, and sampling becomes a permanent engineering decision rather than a setting. You gave up the certainty of the pre-computed answer: a dashboard reading one series is instant and always correct, whereas an ad-hoc query over sampled events is an estimate that needs care to interpret. And you took on instrumentation as ongoing product work, because the system is only as observable as the fields your developers remembered to attach. That is the real trade: predictable cost and known answers, exchanged for the ability to ask something new while production is on fire.
Related Briefs
What to Read Next
- slo and error budget basics
- opentelemetry collector architecture
- high cardinality metrics cost
Observability earns its cost when the questions you need to ask in production outrun the ones you could predict — many services, many deploys, many tenants. Below that, a structured request log, an error tracker and four metrics will resolve your incidents faster and for far less money, and claiming otherwise is marketing. The deciding question is not how much telemetry you collect, but whether you can slice it by a dimension you chose after the incident started.
댓글
댓글 쓰기