How to Set Up Structured Logging in Production
Plain-text logs turn every incident into a grep marathon. Here is how to move to structured logging in production without breaking what already works.
On this page 6 sections
Structured logging means writing each log line as a machine-readable record, usually JSON, with consistent fields such as timestamp, level, service, and request ID. It lets you filter and group events by field instead of grepping text. Set it up by choosing a schema, adding a request ID, and shipping logs to one searchable store.
This guide is for backend and platform engineers who carry a pager and keep hitting the same wall. The logs exist, but nobody can answer a simple question fast. Backtrace covers production tooling for engineers who ship, and this is the setup we would hand a team that is starting from free-text logs. It is an evergreen guide, not tied to a single product release.
Why Plain-Text Logs Fail During an Incident
A free-text log line is written for a human who is already looking at the code. During an outage, the reader is tired and does not know which service is at fault. The line that says "payment failed for user, retrying" tells them almost nothing.
The first problem is parsing. Every team ends up with a pile of regular expressions that extract the user ID, the order number, or the error class from a sentence. One reworded message breaks the dashboard that depended on it, and nobody notices until the next incident.
The second problem is correlation. A single checkout request may touch an API gateway, a payments service, and a queue worker. With plain text, you stitch those lines together by guessing at timestamps. With structured logging, you filter on one request ID and read the whole story in order.
The Google Site Reliability Engineering book chapter on monitoring distributed systems makes a related point. Good monitoring answers what is broken and why, and logs carry most of the "why." Logs that cannot be queried by field answer neither question quickly.
Teams that have moved to structured logs report the same early win. Incident triage stops being a text search and becomes a query. If you have already worked through a big triage queue, such as the one in our Patch Tuesday triage walkthrough, you know how much time disappears into sorting before any real work starts. Logs deserve the same discipline.
Which Fields Does Every Structured Log Line Need?
Every line needs a timestamp in UTC, a severity level, the service name, the build or version, a request or trace ID, and a short event name. Add the user or tenant ID when it is safe to log. These seven fields let an engineer slice an incident by service, release, and customer in seconds.
The event name deserves a closer look. Use a stable, lowercase identifier such as payment_declined rather than a sentence. Dashboards and alerts can then match on the name, and the human-readable detail goes in a separate message field that can change freely.
Here is a small schema that works for most services. The table lists each field, what it holds, and why it earns its place.
| Field | Example value | Why it matters |
|---|---|---|
| timestamp | 2026-10-29T14:03:22.114Z | Sorts events across hosts in one time zone |
| level | error | Lets alerts filter by severity |
| service | payments-api | Isolates the failing component |
| version | 2026.10.3-a1f9c | Ties a spike to a specific deploy |
| request_id | 9c1e7b52 | Joins every line from one request |
| event | payment_declined | Gives dashboards a stable key |
| duration_ms | 412 | Turns logs into latency data |
The OpenTelemetry project defines a standard shape for this data in its documentation on the logs signal. It covers the trace and span identifiers that let a log line link back to a distributed trace. If you adopt those names now, switching vendors later costs far less.
How to Roll Out Structured Logging in Five Steps
The safest path is incremental, one service at a time. Follow these steps in order, starting with the service that pages you most often.
- Pick a logging library that emits JSON natively, such as Pino for Node.js, structlog for Python, or Logback with a JSON encoder for the JVM.
- Define the schema above in one shared module so every service imports the same field names.
- Generate a request ID at the edge and pass it in a header to every downstream call.
- Replace the five noisiest free-text log statements with structured events and compare the old and new queries.
- Ship the output to a single store, then build one saved search per common incident question.
The request ID step is where most rollouts stall. Middleware should read an incoming header such as X-Request-ID, create one if it is missing, and attach it to the logger for the life of the request. In async code, use context-local storage so the ID follows the work across awaits.
Python teams should watch one trap in particular. Context variables can silently drop across tasks if they are set in the wrong place, a pattern we covered in the async/await mistakes that keep showing up in code review. Test that your request ID survives a background task before you trust it.
Do not rewrite every log statement in one pull request. A mixed fleet is fine for a few weeks. Old lines can sit next to new ones as long as the new ones carry the shared fields, and your search tool treats plain text as a fallback.
Verify each step with a real incident question. Pick one from your last postmortem, such as which customers saw errors after a specific deploy. If the new logs answer it in a single query, the schema is good enough to spread. If you still need to read raw lines, add the missing field and try again.
Agree on naming rules early, because they are hard to change later. Use snake_case for every field, keep one spelling for each concept, and never reuse a field name with a different type. A field that is a number in one service and a string in another will break indexing in most log stores. Put these rules in the shared module so a linter or a unit test can enforce them.
What Should You Never Put in a Log Line?
Secrets, full card numbers, passwords, session tokens, and raw personal data should never reach a log. Logs are copied, indexed, and retained for months, and they are often readable by far more people than the database is. A leaked log is a breach with a long tail.
The Open Worldwide Application Security Project (OWASP) keeps a Logging Cheat Sheet that lists what to exclude or mask. It names access tokens, authentication passwords, encryption keys, and sensitive personal data among the items to keep out. Treat that list as a minimum.
Redaction works best when it lives in the logger, not in each call site. Configure an allowlist of fields per event, and drop or hash anything that is not on it. A deny list always misses the new field someone adds next quarter.
Also guard against injection. If a user-controlled string such as a username goes into a log line unescaped, an attacker can forge entries with embedded newlines. JSON encoding handles this for you, which is one more quiet benefit of structured output.
Should You Log Everything at DEBUG Level?
No. Debug logging in production multiplies volume and cost, and it buries the signal you need during an outage. Keep INFO as the default, turn DEBUG on for one service or one request ID when you are chasing a bug, and switch it off again afterward.
Volume is the hidden cost of a logging program. A service that handles 2,000 requests per second and writes ten lines per request produces 20,000 lines per second. At roughly 500 bytes a line, that is about 860 GB per day before compression.
Three habits keep that number sane. First, sample high-volume, low-value events such as health checks, and keep every error. Second, set retention by tier, for example seven days of full logs and ninety days of errors only. Third, review the top ten noisiest event names every month and delete the ones nobody queries.
Infrastructure changes can also shift log volume overnight. A provider upgrade that alters retry behavior can double the error lines from a service, as teams found when planning the Terraform Kubernetes provider 3.4 upgrade. Watch your log volume graph for a day after any such change.
One judgment call is worth stating plainly. Some teams log request and response bodies for debugging. We would not do it by default, because bodies carry the most sensitive data and the largest payloads. Log sizes and schema versions instead, and capture a body only behind a short-lived flag.
Conclusion
Structured logging is a small change with an outsized payoff. The work is mostly agreeing on field names, passing a request ID everywhere, and keeping secrets out. None of it needs a new vendor.
Start this week with your noisiest service. Add the seven fields, ship one saved search for the question you ask most during incidents, and compare how long it takes with and without the new format. Backtrace editors who have run this exercise on a single service consistently find that the first saved search pays for the whole effort.
