Structured Logging and Observability in Backend Systems
Your logs are probably lying to you. Here's how to make them useful.
A few months ago I spent almost four hours tracking down a bug in production. The request was failing intermittently, maybe one in every fifty calls. I had logs. Plenty of them. But they looked like this:
Error: connection refused
Error: timeout exceeded
Info: request completedNo timestamp context. No request ID. No indication of which service, which user, or which downstream dependency was involved. I was reading thousands of lines of text, trying to mentally correlate entries by their position in the file. Like solving a puzzle where half the pieces are from a different box.
That experience changed how I think about logging. Not as an afterthought you sprinkle into your code, but as infrastructure you design up front. The same way you’d design your database schema or your API contracts.
The problem with unstructured logs
Most applications start with console.log or fmt.Println. And honestly, that’s fine for a single process running on your laptop. You can read the output. You know what’s happening because you just wrote the code.
But the moment you have two services talking to each other, unstructured text logs become almost useless. You can’t filter them. You can’t correlate them across services. You can’t aggregate them into dashboards. You end up grepping through gigabytes of text, hoping the timestamp and some keyword will be enough.
The core issue is that unstructured logs are optimized for humans reading a single stream in real time. Production systems don’t work that way. You have dozens of processes, thousands of requests per second, and you’re usually looking at the logs hours after the problem happened.
Structured logging solves this by treating every log entry as a data record with typed fields, not a sentence for a human to read.
What structured logging actually looks like
Instead of this:
User 4521 failed to authenticate: invalid passwordYou produce this:
{
"timestamp": "2025-01-15T14:32:01.442Z",
"level": "warn",
"message": "authentication failed",
"user_id": "4521",
"reason": "invalid_password",
"ip": "192.168.1.42",
"service": "auth-api",
"request_id": "req-a8f3c"
}Same information. But now every field is queryable. You can ask your log aggregator: “show me all authentication failures in the last hour, grouped by reason.” Or: “show me every log entry with request_id: req-a8f3c across all services.” You couldn’t do either of those with plain text.
The format doesn’t have to be JSON. Some teams use logfmt (the key=value format popular in the Go ecosystem). The point is that the log entry has a predictable structure with named fields.
Setting up structured logging in practice
I’ll show this in two languages because the patterns are the same, but the ergonomics differ.
Go (standard library, slog)
Go added the log/slog package in version 1.21. Before that, most teams used third-party libraries. Now there’s a good option in the standard library.
package main
import (
"log/slog"
"os"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
}))
slog.SetDefault(logger)
slog.Info("server starting",
"port", 8080,
"environment", "production",
)
}This outputs a JSON line with timestamp, level, message, and your custom fields. No extra dependencies.
Where slog really shines is when you create child loggers with pre-bound fields:
func handleRequest(w http.ResponseWriter, r *http.Request) {
requestID := r.Header.Get("X-Request-ID")
if requestID == "" {
requestID = generateID()
}
log := slog.With(
"request_id", requestID,
"method", r.Method,
"path", r.URL.Path,
)
log.Info("request received")
user, err := authenticate(r)
if err != nil {
log.Warn("authentication failed", "error", err.Error())
http.Error(w, "unauthorized", http.StatusUnauthorized)
return
}
log = log.With("user_id", user.ID)
log.Info("user authenticated")
// Every subsequent log in this handler carries request_id, method, path, and user_id
}Every log line from this handler now carries the request ID, the HTTP method, the path, and (once authenticated) the user ID. You don’t pass these fields around manually to every function call; you build them up as context accumulates.
Node.js (pino)
In Node.js, pino has been the go-to structured logger for years. It’s fast (it writes logs asynchronously by default) and outputs JSON.
const pino = require('pino');
const logger = pino({
level: process.env.LOG_LEVEL || 'info',
});
logger.info({ port: 8080, environment: 'production' }, 'server starting');The child logger pattern works the same way:
function handleRequest(req, res) {
const requestId = req.headers['x-request-id'] || generateId();
const log = logger.child({
request_id: requestId,
method: req.method,
path: req.url,
});
log.info('request received');
try {
const user = authenticate(req);
const userLog = log.child({ user_id: user.id });
userLog.info('user authenticated');
// use userLog for the rest of this request
} catch (err) {
log.warn({ error: err.message }, 'authentication failed');
res.writeHead(401);
res.end('unauthorized');
}
}The pattern is identical across languages. Create a logger, bind context fields as you learn them, and every log entry automatically carries that context.
Correlation IDs: the thing that makes distributed debugging possible
I mentioned request_id in both examples. This is probably the single most impactful thing you can add to your logging infrastructure. I’m sure about this one.
The idea is simple. When a request enters your system, you generate a unique ID (or read one from an incoming header). Every service that touches this request includes that ID in its logs. When something goes wrong, you search for that one ID and get the full story across every service.
Here’s the thing that trips people up: you have to propagate it. If Service A calls Service B, A needs to include the correlation ID in the outgoing request header. B needs to read it and attach it to its own logger.
// Service A: outgoing call
func callServiceB(ctx context.Context, requestID string) error {
req, _ := http.NewRequestWithContext(ctx, "GET", "http://service-b/data", nil)
req.Header.Set("X-Request-ID", requestID)
resp, err := http.DefaultClient.Do(req)
// ...
}
// Service B: incoming middleware
func correlationMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requestID := r.Header.Get("X-Request-ID")
if requestID == "" {
requestID = generateID()
}
ctx := context.WithValue(r.Context(), "request_id", requestID)
next.ServeHTTP(w, r.WithContext(ctx))
})
}The header name is a convention, not a standard. X-Request-ID is common. Some teams use X-Correlation-ID or X-Trace-ID. Pick one and stick with it everywhere. The name matters less than consistency.
If you’re using a message queue (Kafka, RabbitMQ, SQS), put the correlation ID in the message metadata too. Async boundaries are where correlation breaks down most often, and that’s exactly where you need it most.
Log levels: fewer than you think
I’ve seen logging configs with eight or ten levels. In practice, I use four:
debug is for development. Verbose, noisy, turned off in production unless you’re actively investigating something.
info records normal operations. Request received, request completed, job started, job finished. The “everything is fine” level.
warn means something unexpected happened, but the system handled it. A retry succeeded. A cache miss on something you expected to be cached. A deprecated endpoint got called.
error means something failed and a human should probably know about it. A database query timed out and the request returned a 500. A downstream service is unreachable. Payment processing failed.
I’ve seen teams debate whether they need fatal or critical as separate levels. I think that’s overthinking it. If your process is about to crash, log it as error with a field like ”fatal”: true and let your alerting system pick it up.
The more important discipline is not *which* level to use but making sure you actually use them consistently. I’ve worked on codebases where error was used for input validation failures. That meant the error rate dashboard was always noisy, and real errors got buried. That’s a people problem, not a tools problem, but you can make it easier by writing down what each level means for your team.
What to log (and what not to)
This is where I see the biggest mistakes.
Log too little and you can’t debug anything. Log too much and you can’t find anything (and your log storage bill gets alarming).
Things I always log:
Incoming requests (method, path, status code, duration). Outgoing calls to other services or databases (target, duration, success or failure). Business events that matter (order created, payment processed, user signed up). Errors and the context around them.
Things I never log:
Request or response bodies in production (too much data, and you will accidentally log passwords or tokens). Personally identifiable information without redaction. Secrets, API keys, or database credentials. Health check requests (they flood your logs with noise and they tell you nothing useful).
The PII point deserves emphasis. I’ve seen production logs that included full email addresses, phone numbers, and once even credit card numbers. Structured logging makes this worse because it’s so easy to just spread an object into your log fields. You need explicit allowlists or redaction for any user-facing data.
// Don't do this
log.info({ user: req.body }, 'user signup');
// Do this
log.info({
user_id: user.id,
email_domain: user.email.split('@')[1],
plan: user.plan,
}, 'user signup');From logs to observability
Structured logs are one pillar. On their own, they’re a massive improvement over unstructured text. But they’re not the full picture.
The industry generally talks about three signals: logs, metrics, and traces. I think of it differently. Logs tell you *what* happened. Metrics tell you *how much* is happening. Traces tell you *where time was spent*.
You don’t need all three on day one. Start with structured logs and correlation IDs. That alone will solve 80% of your debugging problems. I’m pretty confident about that number based on my own experience, though your mileage will vary.
When you’re ready to add metrics, the pattern is similar to structured logging: named, typed, labeled data points. Request duration as a histogram, labeled by service and endpoint. Error counts, labeled by error type. Queue depth over time. The same fields you put in your logs (service name, endpoint, environment) should appear as labels on your metrics.
Traces are the most complex to set up. They require instrumentation at every network boundary, and the tooling is still maturing. OpenTelemetry has become the standard way to collect all three signals, and most popular frameworks have integrations for it. But I’ll be honest: I’ve only set up tracing on one production system, and it took significantly more effort than I expected. Worth it for complex distributed systems with many hops. Probably overkill for a two-service architecture.
Operational patterns that actually help
A few things I’ve learned the hard way.
First, always include the service name and environment in every log entry. This sounds obvious, but when you’re aggregating logs from twelve services into one system, and someone forgot to tag their logs with the service name, you’ll spend twenty minutes figuring out where a log line came from.
Second, log at the boundaries. When a request enters your service, log it. When it leaves (response sent), log it with the duration and status code. When you call another service, log the call and the result. If you do nothing else, boundary logging with correlation IDs gives you a timeline of every request through your system.
Third, make your log pipeline resilient. Your application should not crash because the log aggregator is down. Write to stdout, let a sidecar or agent handle shipping. If the agent falls behind, you lose some logs. That’s better than losing the service.
Remember that correlation ID propagation I talked about earlier? It becomes even more important here. When you have boundary logging on every service, the correlation ID is the thread that stitches those boundary events into a coherent story. Without it, you have a pile of disconnected entries. With it, you have a timeline.
Fourth, and I could be wrong about this being universally true, but I’ve found that a single log entry at the end of a request with all the context (duration, status, user ID, any errors) is more useful than many small entries scattered throughout. Some teams call this the “request summary log.” It gives you one searchable record per request, which makes aggregation and alerting much simpler.
The mistake I keep seeing
Teams invest in log aggregation tooling but never invest in log quality. They set up Elastic Search or Loki or whatever, pipe everything in, build dashboards. Then when something breaks, they search the logs and find entries like:
Something went wrong
Error occurred
nullThe aggregation system is only as good as what you put into it. If your log entries don’t have context, no amount of tooling will help. Spend the time making every log entry useful on its own. Include the operation that failed, the inputs that caused it (redacted if sensitive), and what the system did about it.
// This is useless
slog.Error("database error")
// This is useful
slog.Error("failed to fetch user profile",
"user_id", userID,
"query_duration_ms", elapsed.Milliseconds(),
"error", err.Error(),
"retry_attempted", true,
"retry_succeeded", false,
)The second entry tells you everything. Who was affected, how long the query ran, whether a retry was attempted, and whether it worked. That’s the difference between “something broke” and “I know exactly what broke and for whom.”
Where to start if you have nothing
If you’re starting from scratch or retrofitting an existing system, here’s what I’d do in order:
1. Pick a structured logger for your language. In Go, use log/slog. In Node.js, pino is the safe choice. In Python, the standard logging module with a JSON formatter works fine. Don’t build your own.
2. Add a correlation ID middleware. Generate an ID on the first service that receives a request. Propagate it through headers on every outgoing call.
3. Log at every boundary. Request in, request out, external call made, external call returned.
4. Write to stdout. Let the infrastructure handle shipping logs somewhere searchable.
5. Agree with your team on what each log level means. Write it down. Two sentences per level is enough.
That’s it for the first pass. You can add metrics, traces, and fancier tooling later. But these five steps will transform your ability to debug production issues. I’ve done this exact sequence on three different projects now, and each time the improvement was immediate.
The honest truth is that structured logging isn’t exciting. Nobody’s going to write a blog post about how your JSON log lines are a breakthrough. But the first time you search for a correlation ID and see the entire request path across four services in under a minute, you’ll wonder how you ever worked without it.