Post by 
July 21, 2026 | 
The Prometheus, Grafana, and Alertmanager stack has become the default observability foundation for production infrastructure teams over the last decade. Understanding how these components fit together, where each one has hard limits, and where teams consistently make architectural mistakes is the difference between a monitoring system that catches real incidents and one that floods on-call engineers with noise. This guide covers the full stack: the scrape model, the exporter layer, dashboard architecture, alert routing, long-term storage options, and when a managed alternative makes more sense than building your own.
The Modern Open-Source Monitoring Stack: Components and How They Fit Together
The core stack has four distinct layers. Prometheus collects and stores metrics. Exporters expose metrics from targets that do not natively speak the Prometheus format. Grafana visualizes metrics and provides dashboarding. Alertmanager handles alert routing, grouping, and notification delivery.
These components communicate through well-defined interfaces. Prometheus scrapes exporters over HTTP at a configurable interval, typically 15 or 30 seconds. When a configured alerting rule fires, Prometheus pushes the alert to Alertmanager. Grafana queries Prometheus directly using PromQL and renders results into dashboards. Nothing in this chain requires a message broker or a shared database, which is part of why the stack is operationally simple at small to medium scale.
The design intentionally keeps Prometheus stateless from an alerting perspective. Prometheus evaluates rules but delegates all routing decisions to Alertmanager. This separation matters because it lets you change your notification logic, silences, and routing trees without touching the Prometheus configuration or restarting scrapers.
Prometheus: Scraping Model, TSDB, PromQL, Federation, Cardinality, and Retention Limits
Prometheus scrapes targets over HTTP GET requests to a `/metrics` endpoint. Each scrape collects all exposed metrics from that target as a snapshot in time. Prometheus stores these samples in its local time-series database (TSDB), a custom block-based storage engine optimized for high write throughput and sequential reads.
The TSDB compresses data aggressively. A typical sample requires 1 to 2 bytes on disk after compression, compared to the naive 16 bytes per float64 timestamp pair. A Prometheus instance handling 1 million active time series at a 15-second scrape interval writes roughly 50,000 samples per second, and that workload runs comfortably on a 4-core server with 16 GB of RAM for mid-sized deployments.
Cardinality: The Hard Limit You Will Eventually Hit
Cardinality is the number of unique time series in your Prometheus instance. Each unique combination of metric name and label set creates a new time series. A metric like `http_requests_total` labeled by `method`, `status_code`, `endpoint`, and `user_id` can explode to millions of series if `user_id` has high cardinality. Prometheus starts degrading at around 5 to 10 million active series on typical hardware, and query latency gets painful well before that.
The fix is label discipline. High-cardinality labels like user IDs, email addresses, and UUIDs belong in logs or traces, not in Prometheus metrics. If you are seeing scrape durations climb past 10 seconds or queries timeout at 30 seconds, cardinality is almost always the first thing to check.
PromQL and Federation
PromQL is a functional query language built around time-series selection, aggregation, and arithmetic. It is powerful but has real footguns. Range vector selectors, the `[5m]` syntax in queries like `rate(http_requests_total[5m])`, require at least two scrapes within the range window to produce output. Gaps in scrape data produce silent holes in graphs rather than errors.
Federation lets one Prometheus instance scrape a subset of metrics from another Prometheus instance. Teams use this to build hierarchical setups: per-datacenter Prometheus instances feed an aggregate global instance. Federation works for coarse aggregation, but it is not a scaling solution. Pulling pre-aggregated metrics across federation boundaries adds latency and loses the raw series data at the global level.
Retention defaults to 15 days. You can extend this with `–storage.tsdb.retention.time`, but local disk is the ceiling. For anything beyond 30 to 90 days, you need a remote write backend.
The Exporter Layer: Node Exporter, Blackbox Exporter, Process Exporter, and Custom Exporters
Exporters translate metrics from third-party systems into the Prometheus text exposition format. Understanding which exporter to use for which problem is a core operational skill.
Node Exporter
Node Exporter is the standard agent for Linux host metrics. It exposes CPU, memory, disk, network, and filesystem metrics through collectors that read directly from `/proc` and `/sys`. For a deep breakdown of what these metrics mean in production, see CPU Monitoring Explained: How to Read, Interpret, and Act on CPU Metrics in Production, Disk I/O Monitoring Explained: Latency, Queues, and Silent Degradation, and Memory Monitoring Explained: Pressure, Caching, and Failure Modes in Production.
Node Exporter runs as a systemd service on every monitored host. It listens on port 9100 by default. You do not need root privileges to run it, but some collectors require access to privileged files under `/proc`. On containerized workloads, mount the host’s `/proc` and `/sys` into the container to get accurate host-level metrics.
Blackbox Exporter
Blackbox Exporter performs active probing: HTTP, HTTPS, DNS, TCP, and ICMP checks against external or internal endpoints. This is synthetic monitoring from Prometheus’s perspective. You define probe targets in Prometheus scrape configs, and Blackbox Exporter returns success/failure, latency, and TLS certificate expiry for each probe.
For teams monitoring CDN edge nodes or API endpoints, Blackbox Exporter is how you confirm that a service is reachable from outside the host, not just running as a process. This complements DNS Monitoring by giving you response-layer visibility beyond resolution.
Process Exporter
Process Exporter exposes per-process CPU, memory, and file descriptor metrics. It is useful when you need to track specific application processes rather than the whole host. For the broader picture of how exporters and agents compare, see how monitoring agents work on Linux, Windows, and network devices.
Custom Exporters
When no existing exporter covers your application, you write one. The Prometheus client libraries for Go, Python, Java, and Ruby handle the HTTP server and text format serialization. Your code just registers and updates metric values. The most common custom exporter use cases are application-level business metrics (job queue depth, cache hit rate, active sessions) and proprietary hardware or appliances without existing exporters.
Keep custom exporters simple. Each exporter should expose one logical system’s metrics. Combining multiple unrelated systems into one exporter creates operational confusion and makes per-service scrape configuration harder.
Grafana: Dashboard Architecture, Data Sources, and Alerts vs. Alertmanager
Grafana is a visualization layer, not a storage layer. It connects to data sources and queries them on demand when users load dashboards or when alert evaluation runs. Prometheus is the most common data source, but Grafana supports Loki for logs, Tempo for traces, InfluxDB, Elasticsearch, and dozens of others.
Dashboard Architecture
Grafana dashboards consist of panels, rows, and variables. Variables let you build one dashboard that works across multiple environments or hosts. A `$instance` variable populated from a Prometheus label query lets users switch between servers without duplicating dashboard JSON.
Store dashboard JSON in version control. Grafana’s built-in database stores dashboards in SQLite or PostgreSQL, but treating the database as the source of truth creates drift problems. Use Grafana’s provisioning directory to load dashboards from files at startup. This makes dashboard changes reviewable and reversible.
Grafana Alerts vs. Alertmanager
This distinction causes persistent confusion. Grafana has its own alerting system that evaluates queries against data sources and sends notifications. Alertmanager is a separate system that receives firing alerts from Prometheus and handles routing, grouping, and deduplication.
Grafana alerts work well for visualization-adjacent alerts where the condition is directly tied to what you are looking at in a panel. Alertmanager is better for infrastructure alerting where you need sophisticated routing logic: send storage alerts to the storage team, deduplicate alerts from 50 nodes into one notification, and suppress child alerts when a parent system is already known to be down.
For production infrastructure monitoring, Alertmanager should own the critical path. Grafana alerts are a useful supplement for application-level dashboards managed by product teams.
Alertmanager: Routing Trees, Grouping, Silences, and Inhibition Rules
Alertmanager receives alerts from one or more Prometheus instances and decides how to route them. The configuration is a routing tree: a root route with matchers, followed by child routes that match on label values.
Routing Trees
A well-designed routing tree routes by severity and team. A root route might send everything to a general Slack channel, with child routes that match `severity=critical` and send to PagerDuty, and further child routes that match `team=networking` and route to a network-specific receiver. Routes evaluate top-down and stop at the first match unless `continue: true` is set.
The `group_by` field controls which labels Alertmanager uses to group related alerts into a single notification. Grouping by `[alertname, cluster, namespace]` means all pod restart alerts in one namespace arrive as a single notification rather than 20 individual pages. This is one of the most impactful configurations for reducing on-call noise.
Grouping, Wait Times, and Repeat Intervals
Alertmanager introduces three timing parameters that control notification cadence. `group_wait` delays the first notification after a new group fires, allowing related alerts to accumulate. Typical values are 30 to 60 seconds. `group_interval` controls how often Alertmanager sends updates for an ongoing group. `repeat_interval` controls how often a notification resends for an alert that has not resolved. Setting `repeat_interval` too low causes notification fatigue. Setting it too high means you miss unresolved incidents. Most teams land between 1 and 4 hours for production-critical alerts.
Silences and Inhibition Rules
Silences are temporary suppression rules created through the Alertmanager UI or API. You define a silence with label matchers and a time window. Any alert matching those matchers during that window does not generate notifications. Use silences for planned maintenance windows rather than fighting your own alerting system.
Inhibition rules are structural suppressions defined in config. They suppress alerts matching one set of labels when another alert matching a different set of labels is already firing. The classic use case: suppress all disk and network alerts for a host when a `node_down` alert for that host is already active. Without inhibition rules, a single host failure generates dozens of redundant notifications.
Long-Term Storage and Scaling: Thanos, Cortex, and VictoriaMetrics
A single Prometheus instance hits practical limits around 5 to 10 million active series and 90-day retention on reasonable hardware. When you exceed those bounds, or when you need high availability without data loss during restarts, you need a remote storage backend.
Thanos
Thanos extends Prometheus with global query view, high availability, and long-term object storage. The Thanos Sidecar process runs alongside each Prometheus instance, uploads TSDB blocks to object storage (S3, GCS, or Azure Blob), and serves queries from the local Prometheus store. A Thanos Querier federates queries across multiple Prometheus instances with deduplication. Thanos Compactor handles downsampling and retention enforcement in object storage.
Thanos is operationally heavier than plain Prometheus. You are trading deployment complexity for unlimited retention and global query capability. For teams with multiple regions or clusters, it is often the right trade.
Cortex and VictoriaMetrics
Cortex is a horizontally scalable, multi-tenant Prometheus backend. It uses a microservices architecture with separate components for ingestion, querying, and storage. Cortex is the basis for Grafana Mimir and several hosted monitoring platforms. It is powerful but complex to operate: you are essentially running a distributed database.
VictoriaMetrics is a strong alternative for teams that want better performance without the operational complexity of Cortex. It ingests Prometheus remote write, stores data efficiently, and handles cardinality and query loads that would slow Prometheus significantly. VictoriaMetrics publishes benchmarks showing 5 to 10x better ingestion throughput than vanilla Prometheus at equivalent hardware, and its single-node version is operationally simple. For mid-scale deployments that have outgrown Prometheus but want to avoid Cortex’s complexity, VictoriaMetrics is often the right answer.
Managed Alternatives: Datadog, Grafana Cloud, AWS CloudWatch
Building and operating a monitoring stack has a real cost. Before committing to self-hosted Prometheus, understand what the managed alternatives actually cost and what they trade away.
Datadog is the most feature-complete managed option. It handles ingestion, storage, alerting, APM, log management, and synthetics in one platform. The pricing model, per-host plus per-metric per-million custom metrics per month, becomes expensive at scale. A 500-host infrastructure with rich application metrics can run $30,000 to $80,000 per year or more. The tradeoff is zero operational burden for the monitoring infrastructure itself.
Grafana Cloud gives you managed Prometheus, Loki, and Tempo with a Grafana frontend. The free tier covers small deployments. The paid tier uses a consumption model based on series and log volume. For teams already invested in the Prometheus ecosystem, Grafana Cloud avoids the Thanos or Cortex operational work while keeping the same PromQL and dashboard tooling.
AWS CloudWatch is the right choice when your infrastructure is predominantly AWS and your monitoring needs are relatively straightforward. The native integrations with EC2, RDS, ELB, and Lambda remove a lot of setup work. CloudWatch’s query language and alerting are less flexible than PromQL and Alertmanager, and cross-account or multi-cloud visibility requires additional tooling.
The decision comes down to team size, scale, and where you want to spend engineering time. A five-person startup should probably start with Grafana Cloud or Datadog. A 40-person platform team with 2,000 servers and strict data residency requirements should probably own their stack.
What Goes Wrong: Common Prometheus and Grafana Architecture Mistakes
Scrape Intervals Too Short for Target Count
Setting a 5-second scrape interval across 500 targets creates 100 concurrent scrape goroutines every 5 seconds. Prometheus handles this, but the target servers also need to respond to that load. Some exporters, especially custom ones that do expensive operations during scrape, become a bottleneck. Measure scrape duration using `scrape_duration_seconds` and alert when it exceeds 80% of your scrape interval.
Alerting on Raw Counters Instead of Rates
Prometheus counters reset when a process restarts. An alert that fires when `http_errors_total > 100` will miss slow error accumulation and produce false positives after every deployment. Use `rate()` or `increase()` over an appropriate window instead. The window length matters: a 1-minute rate is sensitive but noisy, a 5-minute rate is more stable but slower to detect spikes.
No Alertmanager High Availability
A single Alertmanager instance is a single point of failure for your entire alerting pipeline. Alertmanager supports clustering through a gossip protocol. Running three instances in a cluster with consistent configuration ensures alerts route even when one instance is down. Many teams skip this step and discover the gap during a real incident when alerts stop delivering.
Dashboards Without Ownership
Grafana dashboards accumulate over time. After 12 months, most environments have dozens of dashboards that nobody owns, that use deprecated queries, or that reference metrics that no longer exist. Establish a dashboard ownership model: every dashboard has an owning team in its metadata, and unused dashboards get cleaned up on a quarterly basis.
Forgetting the Network Monitoring Layer
Prometheus-based stacks cover host and application metrics well. Network path visibility between services is harder. Teams often have excellent CPU and memory dashboards and no visibility into inter-service latency, packet loss, or BGP prefix health. Blackbox Exporter covers some of this, but SNMP exporters and dedicated network monitoring tools often need to sit alongside the Prometheus stack for full coverage.
Retention Gaps During Prometheus Restarts
Prometheus writes its WAL (write-ahead log) to disk and recovers from it after a restart. However, recovery takes time proportional to the WAL size. During that window, scrapes are not happening and metrics are not being collected. For high-availability requirements, run two Prometheus instances scraping the same targets and use Thanos or VictoriaMetrics deduplication at query time.
How BlueGrid.io Builds and Operates Monitoring Stacks for Clients
At BlueGrid.io, the NOC team processes over 50 million monitoring events per month across infrastructure clients that include CDN providers, SaaS platforms, and cybersecurity companies. That volume requires a monitoring stack that is both highly reliable and operationally maintainable under real incident pressure.
For clients running self-hosted infrastructure, BlueGrid.io’s systems engineering team designs Prometheus deployments with explicit cardinality budgets from day one. Every label added to a metric requires justification for why that cardinality is necessary. Alertmanager routing trees are designed around team topology, not just technical severity, so the right person receives the right alert at 3 AM without needing to triage it first.
The 24/7 NOC monitoring service integrates directly with client Alertmanager instances or managed infrastructure monitoring tooling. BlueGrid.io NOC engineers receive alerts with full context: the firing rule, the affected host, recent metric history, and the runbook link. That context is built into the alert labels and annotations at design time, not reconstructed during an incident. Response time for P1 incidents stays within a 1-hour SLA because the alert routing and runbook infrastructure does the triage work before the engineer picks up the phone.
For clients that have grown past single-instance Prometheus, BlueGrid.io evaluates VictoriaMetrics cluster deployments and Thanos setups based on data residency requirements, query patterns, and the client’s operational capacity. Not every client needs Thanos. Recommending the simplest architecture that meets the actual requirements is part of the work.
