Logs in Translation

This memo provides information for the curious and the on-call. It does not specify an Internet standard of any kind. Distribution of this memo is unlimited, and comments are welcome by email (saksh.am [at] outlook [dot] in). I don’t trust comment boxes online.

Logs are the most familiar of the three observability signals. This memo follows a log from the application that writes it, through the agent that collects it, to the store that keeps it.

In RFC 0001 we met the three signals. This memo is about the oldest and most familiar of them: logs.

A log starts out as a line of text inside an application and ends up, somewhere else, as a searchable record you read at 3 AM. In between, it gets collected, translated and stored, hence the title. Let’s follow one.

Logs are the language of any observable system. They are a primary source of information about what is happening inside a system. Logs can be structured or unstructured, and they can be generated by applications, services, or infrastructure components.

We use the word “logs” pretty much all the time, but in an effort to provide a clearer definition, we can think of logs as a record of events that have occurred within a system. Note: Not all events are logged, and not all logs are events. Logs can also include information about the state of the system at a particular point in time, such as configuration settings or resource usage.

A typical log journey from your application to the log store looks like this:

Step 0 (because we count from zero, and because this happens before the pipeline even starts): You decide what the application logs. What to log, what to leave out, and what to redact (passwords, tokens, personal data) before it ever leaves the app. This is called instrumentation, and it sets the ceiling for everything downstream: if the app never emits a field, no pipeline can bring it back.

Step 1: The application writes its logs to stdout, stderr or a log file, and carries on with its day. Someone else has to pick those logs up, and that someone is a logging agent.

Ughh, what is a logging agent?

Worry not, let’s get into that first.

A logging agent is a piece of software that pretty much does one thing:
collect logs → transform them → ship them.

Step 2: The logging agent picks up the logs from those files, stdout or stderr. After collecting them for some time, it transforms them and ships them to the log store.

Here is what “transform” looks like in practice. The app writes a plain line like this:

2026-10-01T12:00:08Z ERROR checkout trace_id=4bf92f3577b34da6a3ce929d0e0e4736 Payment failed: database connection timed out token=s3cr3t-abc123

And the agent turns it into this:

{"timestamp": "2026-10-01T12:00:08Z", "level": "ERROR", "service": "checkout", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "message": "Payment failed: database connection timed out token=[REDACTED]", "kubernetes": {"pod": "checkout-7d9f8b6c5-x2kqp", "namespace": "prod"}}

It parsed the text into fields, added the Kubernetes pod and namespace the log came from, and masked the token. (Step 0 should have kept that token out in the first place, but an agent makes a decent second safety net.)

At this point, I would be remiss if I didn’t mention at least a few popular solutions that already do this for you. All OSS ofc :p

A personal favourite is the Fluent family:

• Fluentd: a logging agent which runs as a daemon on your system, with all the perks a daemon has for you! A daemon blog soon, I promise

However, it’s usually not possible to run a behemoth like this in systems that simply lack the resources to do so. In that case, lightweight alternatives are preferred.

• Fluent Bit: a lightweight agent that is often deployed on Kubernetes as a DaemonSet (one per node) or as a sidecar container in a pod. It has a smaller footprint than Fluentd, but still does the job of collecting, transforming and shipping logs.

Step 3: The log store receives the logs and keeps them.

What this means is that you have a centralized place to look at all the logs from your system. Containers are ephemeral, and when one goes away, any logs that only lived inside it go with it. A lost log means losing insight.

The log store takes data from the logging agents and stores it durably, so it is still there when you need it.

Now, a log store isn’t typically your everyday SQL database. It is closer to a search engine. Think Elasticsearch, OpenSearch or Loki, which are built to ingest huge volumes of timestamped text and search through it quickly. (Loki is the odd one out: it indexes only labels and stores the log lines themselves as compressed chunks.) For contrast, a TSDB (Time Series Database) is the usual home for metrics, which we’ll get to.

Example entries:

{"timestamp": "2026-10-01T12:00:03Z", "level": "INFO", "service": "checkout", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "message": "Payment request received"}
{"timestamp": "2026-10-01T12:00:08Z", "level": "ERROR", "service": "checkout", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "message": "Payment failed: database connection timed out"}

Same trace_id on both entries. Hmm, what could that be?

Topic for another RFC! However, tl;dr: trace IDs are used to correlate logs across different services and components in a distributed system. Logs with the same trace ID belong to the same request, so you can follow it end to end.

Step 4: Analyze the logs for insights and troubleshooting.

Once you have the logs in the log store, you can analyze them to gain insights into the behavior of your system and troubleshoot issues. This can involve searching for specific log entries, aggregating logs by service or level, and visualizing log data to identify trends and patterns.

So that’s the journey: the app decides what to say (Step 0), writes it out (Step 1), an agent collects and ships it (Step 2), a store keeps it searchable (Step 3), and finally someone, probably you at 3 AM, reads it (Step 4).

Notice what this pipeline gives you: a detailed record of what happened, one event at a time. What it doesn’t give you is the big picture. How many payments failed this hour? Which service was slow? Where did one request spend its time? You can dig those answers out of logs, but it’s slow and expensive, and that’s what the other two signals are for.

Logs are the most detailed signal and also the most expensive one to keep. What you get out of them is capped by what you chose to log in Step 0, so decide that on purpose.

So are logs enough?

Not quite. Logs tell you what happened. The trace_id from the example entries is the thread that ties them to traces, and metrics answer the “how much” and “how often” questions far more cheaply. Both are topics for the next RFCs. Yes, I’m promising again

[RFC0001] Saksham, “Observability, My Dear Watson”, blogs.saksh.fyi/observability-my-dear-watson.

[OTEL-LOGS] OpenTelemetry, “Logs”, opentelemetry.io/docs/concepts/signals/logs.

[OTEL-INSTR] OpenTelemetry, “Instrumentation”, opentelemetry.io/docs/instrumentation.

[TRACE-CTX] W3C, “Trace Context”, w3.org/TR/trace-context.

[FLUENTD] Fluentd Project, fluentd.org.

[FLUENT-BIT] Fluent Bit Project, fluentbit.io.

[LOKI] Grafana Labs, “Grafana Loki documentation”, grafana.com/docs/loki/latest.