Reading the Sky: Observability as Wayfinding

There’s a difference between knowing a system is down and knowing where you are inside it. The first is an alarm. The second is navigation. Most teams have plenty of the first and almost none of the second, and they find out which one they have at the worst possible moment.

Observability, stripped of the vendor language, is just this: can the system tell you where you are without you having to guess? When the answer is yes, an incident is a heading correction. When the answer is no, it’s a search party.

Three instruments, one position

Sailors didn’t fix their position from a single instrument. They cross-checked — compass against stars against the log line trailing in the water. No single reading was trustworthy alone; the agreement between them was. Telemetry works the same way, and the three classic signals each answer a different question:

• Metrics — is something wrong? Cheap, aggregate, always on. Your horizon line. Good for “the error rate just tripled,” useless for “why.”
• Traces — where is it wrong? The path of a single request across every service it touched. Your compass bearing. This is the one teams skip and the one they most wish they had at 3am.
• Logs — what exactly happened? The detail at a point you already suspect. Your log line in the water. Precise, expensive, and worthless if you’re reading the wrong spot.

Have one and you’re guessing. Have all three and you can triangulate. The skill isn’t collecting them — it’s reading them together.

Instrument the boundaries, not the lines

The temptation is to log everything. Resist it. The signal that matters lives at the seams — the moments a request crosses from one system into another, where your assumptions meet someone else’s reality.

using var activity = Tracing.Source.StartActivity("charge.payment");
activity?.SetTag("order.id", order.Id);
activity?.SetTag("amount.cents", order.TotalCents);

var result = await _gateway.ChargeAsync(order);

activity?.SetTag("charge.outcome", result.Status);
if (!result.Ok)
    activity?.SetStatus(ActivityStatusCode.Error, result.Error);

One span at the boundary tells you the order, the amount, the outcome, and — via the trace it belongs to — everything that led here and everything that followed. That’s worth a thousand log.debug("entering method") lines that only prove the code ran, which you already knew.

Name things for your future, panicked self

Telemetry is written calm and read terrified. So we name spans, metrics, and log keys for the version of us reading them mid-incident, with adrenaline up and patience gone:

charge.payment with an order.id tag beats ProcessOrderInternalV2. The first answers a question. The second is a question.

The test we run before shipping isn’t “did we add telemetry?” It’s: could a teammate who has never seen this code find the failing request in under five minutes? If not, we haven’t instrumented the system — we’ve just decorated it, and we’ll pay the difference the first night it breaks.

The quiet payoff

Done well, this changes the texture of being on call. The system stops being a black box you poke at and becomes a place you can read. You open the trace, you follow the bearing, you find the spot. You don’t navigate by staring at the ground and hoping. You look up, take a fix from the instruments, and you already know which way is north.