Your app is still responding, but checkout feels slow. The health check passes, and average response time barely moves. Then you look at p95 and see that it has tripled. Some customers are waiting much longer than usual.
Now you have a better question. Where is that time going?
Observability helps you answer questions about a running system using the signals it emits. Metrics show patterns across many requests. Traces follow individual requests through the operations they trigger. Logs record events that help explain what happened. Alerts bring changes to your attention when they need a response.
Metrics show how widespread the problem is
A metric is a measurement you can track over time. For checkout, start with requests per minute, the percentage that fail, and how long requests take. Together, they show whether checkout is slow, whether customers are seeing errors, and how much traffic you’re handling.
Suppose checkout p95 rises from 400 milliseconds to 1.2 seconds while p50 stays near 200 milliseconds. The typical request is still fast, but the slower end of the distribution has shifted. Check how many requests the chart represents and compare checkout with other routes. A slowdown concentrated on one route gives you a smaller area to investigate. Understanding Latency: What p50, p95, and p99 Tell You About Your App
If several routes slow down together, compare application latency with CPU usage, memory pressure, and disk read and write latency over the same time window. Those measurements can help you decide where to look next. Why Your App Is Slow: Understanding CPU, RAM, Disk, and Network
Traces show where a request spent its time
Open a slow checkout trace from the affected window. A trace breaks the request into timed operations called spans. You might see the app validate the cart, look up the customer, call a payment provider, and write the order to the database.
Say the payment operation took 900 milliseconds. Compare that trace with a few typical checkout traces. If you see the same delay across several slow requests, you have a specific operation to investigate. The span shows how long the app spent on payment. The logs can help explain what happened during that time.
Logs give you the details behind the wait
Find the log entries for the same trace or request ID. You might learn that the first payment attempt timed out and the retry succeeded. That explains how customers could wait longer while checkout still returned a successful response.
Logs are easier to search when they carry consistent fields such as the route, service name, trace ID, and error type. An entry that says payment attempt timed out gives you a specific event to look for across other slow requests. Keep credentials and payment details out of the logs.
Alerts tell you when to look
A checkout alert might fire when p95 stays above a threshold for several minutes and enough requests have come through to make the measurement meaningful. Choose a threshold that reflects how long customers can reasonably wait and how quickly someone needs to respond.
A useful alert names the affected operation, gives the time window, and links to the relevant chart or investigation view. In this example, it sends you straight to checkout latency. From there, you can find a slow trace and check the logs for a timeout or retry. If retries become a recurring problem, chart their rate alongside latency so you can see when the two rise together.
Start with one request that matters
Pick a flow your users depend on, such as signing in, checking out, or uploading a file. Track how often it runs, how often it fails, and how long it takes. Trace its main operations and make sure you can find the logs for a single request. Then decide what change should trigger an alert and where it should send the person investigating.
Try it before an incident. Take a real request, find it in a trace, and look up the corresponding logs. Then check the metrics for that time window. Any step you struggle to take tells you where to improve your instrumentation.
Comments