Meta: Prometheus alerting best practices reduce false positives and catch real incidents fast. Learn how to write alerts that fire on user impact, not just internal metric thresholds.

Prometheus Alerting Best Practices for SRE Teams

Prometheus alerting is one of the most widely used alerting approaches in cloud-native environments, and also one of the most commonly misconfigured. The flexibility of PromQL enables precise, nuanced alerting rules—but the same flexibility enables poorly written rules that generate massive alert noise with minimal reliability benefit.

This guide covers the specific practices that produce high-quality Prometheus alerting: alerts that fire when users are impacted, don't fire when they're not, and generate the right escalation at the right severity.

The Foundation: Alerting on Symptoms, Not Causes

The single most important principle in Prometheus alerting is this: alert on user-visible symptoms, not internal system metrics.

An alert that fires when CPU utilization exceeds 80% is alerting on an internal metric. An alert that fires when the p99 latency of your checkout endpoint exceeds 2 seconds is alerting on a symptom. The difference matters because:

High CPU doesn't always cause user impact. A batch job that consumes a lot of CPU might briefly spike CPU without affecting user-facing latency. High CPU that causes request queuing does cause user impact—but the correct alert is on the latency symptom, not on the CPU cause.

Conversely, user-facing latency spikes can be caused by many things other than high CPU: database slowness, network latency, a slow external dependency. If you only alert on CPU and latency is degraded for a different reason, you miss the incident.

Alert on what users experience—error rates, latency percentiles, availability—and use metrics like CPU and memory for investigation, not for alerting. This is the core principle of the Google SRE book's alerting philosophy (the "Four Golden Signals": latency, traffic, errors, saturation) and it produces dramatically lower false positive rates in practice.

Writing PromQL Alert Rules That Work

Prometheus alert rules follow a standard format: a PromQL expression, a threshold, a duration, and labels and annotations.

Use rate() for counters, not instant values: Counters in Prometheus (like http_requests_total) only go up. To measure the rate of requests per second, use rate(http_requests_total[5m]). Using a counter directly in an alert expression produces meaningless alerts.

Choose the right time window for rate(): The time window in rate() controls smoothing. Short windows (1m) are more sensitive to sudden spikes but noisier. Long windows (30m) smooth out spikes but are slower to detect. For most latency and error rate alerts, 5m is a reasonable starting point.

Alert on percentile latency, not average: Average latency is easily skewed by a small number of very fast or very slow requests. Percentile latency (p95, p99) reflects the experience of your slowest users, which is usually the relevant metric for user impact. Prometheus histograms with histogram_quantile() are the standard approach.

Use for duration to avoid flapping: An alert rule with for: 5m only fires if the condition is true for 5 continuous minutes. This prevents brief spikes from generating pages. Choose the duration based on how long the condition needs to persist before it represents a real problem—not how quickly you want to know about it.

Label alerts for routing and context: Alert labels in Prometheus drive routing in Alertmanager. Label severity: critical vs severity: warning routes to different notification channels. Label service: payments associates the alert with the right team. Consistent labeling is what makes alert routing work cleanly.

Write helpful annotations: The summary annotation should describe the alert in plain language. The description annotation should provide the context engineers need to start investigating: which metric, what threshold, what the current value is. Include a link to the relevant runbook and dashboard. Engineers who receive an alert should know what it means without having to look anything up.

Example of a well-constructed alert:

alert: CheckoutHighErrorRate
expr: |
  sum(rate(http_requests_total{service="checkout", status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total{service="checkout"}[5m])) > 0.01
for: 5m
labels:
  severity: critical
  service: checkout
  team: payments
annotations:
  summary: "Checkout service error rate above 1%"
  description: "The checkout service error rate has exceeded 1% for 5+ minutes. Current value: {{ $value | humanizePercentage }}. Runbook: https://wiki.example.com/runbooks/checkout-high-error-rate"

The Golden Signals as Alert Architecture

The Google SRE golden signals—latency, traffic, errors, saturation—provide a useful framework for organizing alert coverage without over-alerting.

Latency alerts: Alert when latency at a meaningful percentile exceeds user-facing thresholds. Define what "too slow" means for each service based on user expectations and SLOs.

Error rate alerts: Alert when the error rate for user-facing endpoints exceeds a defined threshold. For most user-facing services, a 1% error rate is a meaningful signal. Adjust based on your baseline and SLO.

Traffic alerts: Alert when traffic drops significantly below baseline (indicating that users can't reach your service) or spikes above capacity limits. Anomaly-based alerting is more effective than fixed thresholds for traffic, since traffic patterns vary significantly by time of day and day of week.

Saturation alerts: Alert when resource utilization reaches levels that indicate approaching capacity. The key is alerting early enough to respond before saturation causes degradation. For most stateful resources (database connections, queue depth), alert at 70-80% of capacity.

Covering all four golden signals for each service provides complete alert coverage without redundancy.

Alertmanager Configuration for Effective Routing

Prometheus alerting only fires; Alertmanager handles routing, grouping, and notification. Good Alertmanager configuration is as important as good alert rules.

Grouping: Alertmanager groups related alerts and sends them as a single notification. Configure grouping by service and alertname to prevent a single failure from generating dozens of individual pages. See what is alert fatigue for why this matters.

Inhibition: Inhibition rules suppress alerts when a more critical alert is already firing. If a ClusterDown alert is firing, inhibit all per-service alerts from that cluster—the per-service alerts are implied by the cluster-level alert and add noise without information.

Routing trees: Alertmanager routes alerts based on label matchers. Define a routing tree that sends severity: critical alerts to the on-call rotation immediately, severity: warning alerts to a Slack channel for async review, and severity: info alerts to a log for trend analysis.

Silence management: Alertmanager silences suppress alerts for defined periods. Use silences for planned maintenance windows—not as a workaround for noisy alerts that should be fixed.

Common Prometheus Alerting Mistakes

Alerting on raw gauge values without context: An alert that fires when memory usage exceeds 2GB is meaningless without knowing the total memory available. Alert on percentage or utilization ratio, not raw values.

Setting for duration too short: A for: 0s alert fires on the first scrape that crosses the threshold, including brief spikes from transient events. Most alerts should have at least a 2-5 minute duration.

Missing the / for rate: increase(counter[5m]) > 100 counts total increases over 5 minutes. rate(counter[5m]) > 20 measures per-second rate. Mixing these up produces incorrect thresholds.

Not testing alert expressions before deployment: PromQL expressions can have syntax errors or unexpected behavior (like returning no data for some label combinations). Test expressions in Prometheus's expression browser before deploying as alerts.

Alerting on everything by default: Monitoring tools that auto-generate alerts for every metric create immediate alert fatigue. Every alert should be manually reviewed and have an explicit justification for why it represents actionable user impact.

How Fluidify's Agentic Reliability Suite Connects to Prometheus Alerting

Fluidify is an AI SRE suite—or more precisely, what we call an Agentic Reliability Suite—that ingests Prometheus metrics as a core input for its root cause analysis and incident management capabilities.

Regen receives Prometheus alerts via Alertmanager webhook integration. When a Prometheus alert fires, Regen classifies its severity, routes to the appropriate on-call engineer, groups related alerts from the same underlying failure, and creates an incident channel with context—all automatically.

Neuri, Fluidify's Adaptive RCA Engine, uses Prometheus metrics as one of its primary evidence sources for root cause analysis. When an incident is created from a Prometheus alert, the Adaptive RCA Engine automatically queries relevant metrics—comparing current values against historical baselines, correlating metric changes across related services, and identifying the metric anomaly pattern that best characterizes the failure.

Gills, the Natural Language Interface to your stack, allows engineers to query Prometheus metrics in plain language during active incidents. "What's the p99 latency on the checkout service compared to yesterday at this time?" or "Which services have error rates above 1% right now?" return immediate answers without writing PromQL.

Reflex, the Auto Heal Engine, executes remediations when the Adaptive RCA Engine confirms a root cause based on Prometheus metric evidence. For incidents where the metric evidence is clear and the remediation pattern is known, Reflex closes the incident autonomously.

FAQ

What are the best practices for Prometheus alerting? The core best practices are: alert on user-visible symptoms (latency, error rates, availability) rather than internal metrics; use the four golden signals as an alert architecture framework; use for duration to avoid flapping; write descriptive annotations with runbook links; configure Alertmanager grouping to prevent cascade floods; and test alert expressions before deployment.

What is the difference between alerting on a counter vs. a gauge in Prometheus? Counters only increase and represent cumulative totals (total requests, total errors). To alert on counter-based metrics, use rate() to compute the per-second rate. Gauges represent current values (CPU utilization, queue depth) and can be used directly in alert expressions.

How do you prevent false positive alerts in Prometheus? Use for duration to require the condition to persist before firing, alert on percentile latency rather than average to avoid spike sensitivity, use inhibition rules to suppress downstream alerts when a root cause alert is firing, and alert on user-visible symptoms rather than internal thresholds.

What is Alertmanager and how does it work with Prometheus? Alertmanager receives alerts from Prometheus, applies routing rules to send them to the right recipients, groups related alerts to prevent cascade floods, applies inhibition rules to suppress redundant alerts, and manages silences for maintenance windows.

How many alerts should a service have? There's no fixed number, but a well-instrumented service typically has 5-10 alert rules covering the golden signals at different severity levels. Fewer than 3 usually indicates incomplete coverage; more than 15 usually indicates some alerts are alerting on internal metrics rather than user impact.

Connect your Prometheus alerts to AI-driven incident response that diagnoses and remediates automatically. See how Fluidify integrates with Prometheus →