Monitoring Cloud Servers: Metrics, Logs and Alerts

Begin With the Question, Not the Dashboard

It is easy to install a monitoring stack and end up with forty graphs nobody looks at. Useful monitoring starts from questions instead: is the service responding, is it getting slower, is anything about to run out, and if it broke at 04:12 what does the evidence say?

Every graph and every alert should trace back to one of those. If it does not, it is decoration, and decoration is what trains people to ignore the screen.

This guide walks through the three kinds of signal, what to collect from a Linux server, how provider tooling and self hosted stacks compare, and how to write alerts that survive contact with a tired human.

Three Kinds of Signal, Three Different Jobs

Metrics are numbers sampled over time: processor use, memory, request rate, queue depth. They are cheap to keep for a long while, good at showing trends, and the natural basis for alerting.

Logs are discrete events with detail attached. They are expensive at volume and unbeatable for answering why a specific request failed. Metrics tell you something broke; logs usually tell you what.

Traces follow one request across services, showing where the time went. They matter once a request touches more than two or three components, and they are usually the last of the three a team adopts.

A small deployment can run well on metrics and logs alone. Adding tracing before you have reliable alerting is optimising the wrong end of the problem.

What Is Actually Worth Collecting From a Server

For each resource on the machine, three numbers tell the story: how busy it is, how much work is waiting, and how often it failed. That framing keeps the list short.

Use percentiles rather than averages for latency. An average response time can look healthy while a meaningful share of requests time out, because a handful of fast responses drown the slow ones.

Add one external check that fetches a real URL from outside the provider network. It catches the failures that in host monitoring cannot see, such as a firewall change, an expired certificate or a DNS record pointing at a decommissioned address.

Provider Tooling or Your Own Stack

Every major platform ships something. AWS has CloudWatch with its unified agent for in guest metrics, Azure has Azure Monitor, Google Cloud has Cloud Monitoring, and developer clouds such as DigitalOcean, Linode and Hetzner expose basic instance graphs plus a lightweight agent. The CloudWatch documentation is a good model of what the native layer typically includes.

Native tooling wins on integration. It already knows your instances, it can alert on platform level events an agent cannot observe, and there is nothing to keep patched. It loses on portability and on cost once custom metrics and long retention arrive.

The common self hosted answer is Prometheus scraping a node exporter on each machine, Grafana for dashboards, Alertmanager for routing, and Loki or an Elasticsearch based stack for logs. Prometheus is a graduated CNCF project and its query language is worth learning even if you end up paying someone else to run it.

Mixed estates usually settle on the self hosted route, because one dashboard across providers beats three portals. Single provider teams often stay native for longer than they expect to.

Getting Logs Off the Box

Logs stored only on the instance disappear exactly when you need them, because the instance that failed is often the instance you rebuild. Ship them somewhere else, and do it before the first incident rather than after.

On a systemd host, the journal is the collection point for almost everything. An agent such as Promtail, Vector or Fluent Bit reads it, attaches labels for host and service, and forwards to a central store. Containers write to standard output and the runtime's logging driver does the same job.

Two settings prevent most log related outages. Cap local retention with journald limits or log rotation so a chatty service cannot fill the disk. Then apply a retention policy centrally, because indexed log storage is usually the largest line in a monitoring bill.

Structure the output where you can. JSON lines with consistent field names make filtering trivial, while free text requires a fragile parser that breaks the next time somebody rewords a message.

Alerts Nobody Mutes

An alert should mean a human needs to act now. Anything else belongs on a dashboard or in a weekly review. Applying that single rule usually cuts a noisy alert set in half.

Alert on symptoms rather than causes. "The error rate has been above the threshold for five minutes" is actionable; "processor use hit ninety percent" is often a healthy machine doing its job. High resource use only matters when it shows up in what users experience.

Give every rule a duration so a brief spike does not page anyone, and route by severity: paging for things that need waking someone, a chat channel for things that can wait until morning. Write a one line runbook into the alert description, because the person who receives it at four in the morning may not be the person who wrote it.

Finally, alert on the monitoring itself. A silent dashboard looks identical to a healthy one. A heartbeat check that fires when metrics stop arriving is the cheapest insurance in the whole setup.

Observability Has a Bill

Every custom metric, every log line and every high cardinality label costs storage and query time. Labels that include a request identifier or a user identifier are the classic mistake, since they multiply time series until the database struggles.

Set retention deliberately per signal. High resolution metrics for recent history, downsampled for long term trends, and shorter retention for verbose debug logs than for security relevant events. The trade offs are the same ones described in cloud cost monitoring and budget alerts, and the retention rules for security records specifically are covered in audit logging in the cloud.

Monitoring data also feeds decisions beyond incidents. Scaling rules read the same metrics, as explained in autoscaling explained, which is a good reason to get the collection layer right before automating anything on top of it.

If you are choosing where to run this, note how much the native layer gives you for nothing. The AWS account listings describe an account with the full CloudWatch surface available, and the AWS versus Google Cloud comparison looks at how the two present metrics and logs differently.

A Sensible Starting Point

• Collect utilisation, saturation and errors for processor, memory, disk and network.
• Track latency as percentiles, never as an average.
• Add one external check that hits a real URL from outside the provider.
• Ship logs off the instance and cap local retention so disks cannot fill.
• Alert on user visible symptoms, with a duration and a runbook line.
• Add a heartbeat alert so silence is distinguishable from health.
• Keep label cardinality low and set retention per signal type.
• Review the alert set periodically and delete anything that never led to action.