Observability, My Dear Watson

This memo provides information for the curious and the on-call. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited, and comments are welcome by email (saksh.am [at] outlook [dot] in). I don’t trust comment boxes online.

Ask a software engineer what observability means and you will get a list of tools. This memo argues that a list of tools is not a definition. It explains what observability is, introduces the three signals (logs, metrics and traces), and makes the case for why we need them. Each signal, and OpenTelemetry, gets its own memo in this series.

What do people actually mean when they say “observability”?

To a software engineer, it brings to mind a pile of tools and buzzwords: log aggregation, Prometheus, Grafana, OTel, and a million others. But a list of tools isn’t a definition. If anything, it just raises more questions.

In this post, we’ll cover what observability is, meet the three signals (logs, metrics and traces) at a high level, and look at why we need observability in the first place. Each signal, and OpenTelemetry (OTel), gets its own post in this series.

When talking about systems, observability is the ability to understand what’s happening inside a system based on the data it produces. It’s about being able to ask questions about the system and get answers from the data it generates.

How is this different from monitoring? Well, to put it simply, monitoring answers the questions you thought of in advance, while observability lets you ask new ones.

Saksham, what do you mean by that?

Well, glad you asked!

In a typical monitoring stack, you decide ahead of time what to watch. You might have alerts that notify you when a threshold is crossed (think: latency, high CPU usage). That works for the failures you predicted, but the alert can only notify you, not really help you understand why it happened. This is where observability comes in.

Let’s remove the word for a moment, since I personally believe a lot of the issue in understanding observability is the lack of a clear definition.

Scenario 1: Something degrades, think slow responses. Well, you’d probably have a look at the server logs, the resource consumption to see whether the CPU can keep up with the requests. Now say only the checkout requests from one region are slow. Nobody built a dashboard for that, because nobody thought to ask.

Scenario 2: Something else goes down, think applications failing with an “unable to acquire db connections” error. You’d probably go have a look at the database open connections, the application connection pool, and so on.

In both scenarios, you are trying to understand the system and why it is failing. You are trying to observe the system and understand it. This is observability. And the key part is that you didn’t have to know the question ahead of time.

Observability comes from the English word “observable”, which means “able to be observed”. :P

Observability data usually comes in three flavours, often called the three signals: logs, metrics and traces. Each one answers a different kind of question.

Logs are a record of events: what happened, and when. They are the most familiar signal and the first thing most of us reach for when something breaks, like “Payment failed: database connection timed out”. We follow a log from the application all the way to the store in RFC 0002.

Metrics are numbers measured over time, like CPU usage, requests per second or latency. They are cheap to store compared to logs and great at telling you that something is wrong, which is why most alerts run on them. They won’t tell you why, though.

Traces follow a single request as it travels through your services, showing where the time went and where it failed. Logs tell you what happened inside one service and metrics tell you something is off overall, but traces connect the dots between services.

No single signal is enough on its own, and the interesting part is how they work together. That is where OpenTelemetry (OTel) comes in, and it gets the last memo in the series.
Metrics, traces and OTel memos coming soon.

Do I even need observability? Well, not always.
If you are running a simple application that does one thing, and it is not critical, you might be fine without it. But if you are running a complex system with multiple services, observability is something you would end up needing, sooner rather than later.

Unknown unknowns:
You can only know what you know. You can only alert on failures you have already imagined, and in a complex system even small changes can cause behavior nobody predicted. Your data has to be rich enough to answer questions you didn’t know you’d ask.

Cost of downtime:
When something breaks, most of the time goes into figuring out what broke, not fixing it. The fix is often a restart or a one line change. The hunt is what costs you.

Distributed systems:
One request from a user can go through a bunch of services. When it fails, the clue could be in any of them, and you end up opening one service’s logs after another.

Logs alone don’t scale:
Logs are great, but searching through them gets slow and painful as the system grows. You need something that tells you where to look first.

Fewer 3am pages:
If the answer is in the data, you don’t have to wake someone up just to ask how this thing works.

Understanding, not just fixing:
It’s not only for when things break. You also get to see how the system behaves on a normal day, like how much traffic it takes and where it is slow.

At the end of the day, keeping systems stable is the goal.
Even “perfect” systems break. Being able to look in one place and know exactly what to fix? Sounds like a blessing to me.

Conclusion? Phew! That’s a lot of words. Here’s the TL;DR:

Observability is the ability to understand what’s happening inside a system based on the data it produces. It is about being able to ask questions about the system and get answers from the data it generates. The three signals are logs, metrics and traces, and they work together to give you a complete picture of the system. Observability is not always necessary, but it is essential for complex systems. It reduces downtime and makes it easier to fix problems when they arise.
Observability is not a tool, it is a practice. It is about understanding the system, not just fixing it.

For a website that’s pretty much me fanboying over technology, I would like to thank every one of the people who have contributed to the open source community, and the people who have written about observability and the three signals. You are all awesome.
Here are some references that I found useful in writing this memo:

• Monitoring Distributed Systems - A chapter of the Google SRE book, free to read online
• Observability primer - From the OpenTelemetry documentation
• charity.wtf - Charity Majors’s blog, with many essays on observability and on-call