Observability-Driven Development: Ship, Watch, Learn, Repeat
SLOs, structured logs, high-cardinality metrics, and the discipline of designing every feature with 'how would I debug this at 3am' as a first question.
Observability is not a monitoring dashboard you add after launch — it's a design constraint you carry into every feature. Structured logs, a handful of SLOs, high-cardinality tracing, and the habit of asking 'how would I debug this at 3am' during code review turns firefighting into engineering.
- Every feature is designed for its worst on-call moment, or it isn't ready.
- Structured logs (JSON, one line per event, with a request ID) beat any log format from the past 20 years.
- SLOs, not vanity metrics, tell you whether the system is healthy.
- High-cardinality metrics (user_id, tenant_id) are essential — pick tools that don't punish you for them.
- The best observability is the one your team actually uses in incidents, not the fanciest one.
The 3am test
Every feature you ship will fail at 3am one day. The question during code review is: could a tired on-call engineer, who didn't write this code, figure out why in 15 minutes using only your logs, metrics, and traces? If not, the feature isn't done.
This is not a slogan. Adopt it as a review gate and the shape of your code changes. You start writing structured logs at every branch. You start emitting metrics with useful labels. You stop swallowing exceptions. Bugs that took days to diagnose start taking minutes.
Structured logs, boring and correct
A log line is a JSON object. Every log line has: timestamp, level, event name, request ID, and any context fields relevant to the event. That's it — no clever format, no ANSI colors, no multi-line stack traces mixed with regular logs.
log.info('order.paid', order_id=o.id, customer_id=o.customer_id,
amount_cents=o.total, currency=o.currency,
request_id=ctx.request_id, latency_ms=ms)With this shape, every log is a queryable event. 'Show me all paid orders over $500 from tenant X in the last hour that took more than 2 seconds' is a single query in Loki, Datadog, or BigQuery. Try that with printf-style logs and you'll be there all afternoon.
One request ID, threaded end-to-end
A single request touches your API gateway, several microservices, the database, maybe a background worker. Generate a request ID (or accept one from an upstream) at the entry point and pass it through every log line, RPC call header, and job payload.
When something breaks, the request ID is the string you paste into your log tool to see the entire causal chain. It is the single highest-ROI observability practice and it costs almost nothing to add.
SLOs beat 'is it up?'
Uptime is a lie. What matters is: what fraction of user-visible operations succeeded within acceptable latency in the last 30 days? That's a Service Level Objective.
For each critical endpoint, define an SLI (measurable quantity, e.g. 'p95 latency for POST /checkout under 500ms') and an SLO (target, e.g. '99.5% of requests over 30 days'). Compute the error budget (100% - SLO). Alert on burn rate, not raw error counts:
- Fast burn (2% of monthly budget in 1h) — page immediately.
- Slow burn (10% of budget in 6h) — ticket, next business day.
- Below burn — no alert. The system is behaving as designed.
This alerting philosophy — from Google's SRE book, battle-tested for 15+ years — eliminates false-positive pages and forces you to have real conversations about how reliable you want to be.
High-cardinality is the point
The old advice was to keep metrics low-cardinality (few label combinations). That was a limitation of old TSDBs. Modern observability tools (Honeycomb, Grafana Cloud, ClickHouse-backed engines) handle millions of unique series and give you a superpower: filtering metrics by user_id, tenant_id, feature flag, deploy_id, region.
Insist on high-cardinality-friendly tooling. When production is on fire and the question is 'is this affecting one big customer or everyone?', the answer must be one dashboard-click away.
Traces for the paths logs can't show
A trace shows the causal graph of a single request across services with per-span timings. It answers 'where is the latency actually spent?' better than any log. Adopt OpenTelemetry, instrument your framework's HTTP client and server, and you get 80% of the value with a few days of setup.
Full sampling is expensive; head-based sampling (10-20%) plus tail-based sampling (always keep errors and slow requests) is the pragmatic setup that catches what matters without breaking the budget.
The tools question
Pick one stack, know it deeply. My default for a mid-sized team in 2026:
- Logs: Grafana Loki or Datadog Logs — structured JSON, retention 14-30 days for hot, cheaper archive for cold.
- Metrics: Prometheus or Grafana Cloud — high-cardinality-friendly, SLO burn-rate alerts.
- Traces: OpenTelemetry SDK, exported to Tempo, Honeycomb, or Datadog APM.
- Alerts: PagerDuty or Opsgenie, routing rules by SLO burn rate and service ownership.
- Runbooks: markdown in the same repo as the code, linked from every alert.
The best stack is the one your team looks at during an incident without hesitation. That takes weeks of practice — schedule chaos game days and mock incidents to build the muscle.
Frequently asked
Isn't this overkill for a small team?
Structured logs and a request ID cost you nothing and pay back the first time production misbehaves. Skip elaborate tracing until you have real multi-service latency questions.
How many SLOs should I have?
One or two per critical user-facing endpoint. A team with 40 SLOs has zero SLOs — nobody looks at them.
Traces or logs, if I can only afford one?
Structured logs with a request ID. They cover 80% of what tracing does at 10% of the setup cost. Add tracing when you feel the specific pain of 'where did the time go across five services?'.
Building something similar?
I help teams ship production-grade AI agents, n8n workflows, and data platforms. Let's talk about what you're building.
Work with meIslam Gamal
AI, Data & Automation Engineer. I design and ship production AI agents, n8n workflows, and cloud data platforms — with a focus on reliability, cost, and measurable business impact. Founder of Tashghil and Tek bil Arabi.
More from Islam Gamal
Postgres for Product Engineers: Indexing, JSONB, RLS, and the 20% That Ships Millions of Users
The Postgres playbook I've used across product teams — index type per workload, JSONB without regret, RLS that doesn't leak or crawl, connection pooling that survives spikes, and the observability that turns guessing into ground truth.
Observability for AI Systems: Traces, Prompts, Tokens, and the SLOs That Actually Matter
A 30-minute production guide to instrumenting LLM applications — what to log (and what never to), how to trace agent calls end-to-end, the four golden signals for AI, drift detection on outputs, and the dashboards you actually check at 2 a.m.
Async Python for AI Pipelines: Concurrency, Backpressure, and Not Melting Your API Bill
A pragmatic 30-minute deep-dive into asyncio for AI workloads — semaphores, timeouts, retries with jitter, streaming responses, connection pooling, and the patterns that let one Python process fan out to thousands of concurrent LLM calls without exploding.
Browse every article by Islam Gamal on the author page.