How Monitoring Agents Collect Data: Pull vs Push, Exporters, and Overhead

Monitoring agents are the data collection layer that everything else depends on. Get the architecture wrong – wrong model, wrong scrape interval, wrong agent for the platform – and you produce either data gaps that hide incidents or metric floods that choke your storage. This guide covers how pull and push collection models actually work, where exporters fit into that picture, what agents cost in real resource terms, and what happens when agents fail silently. The BlueGrid.io NOC processes more than 50 million monitoring events per month, so the failure modes described here are drawn from production infrastructure, not from documentation.

Agent vs Agentless Monitoring: The Actual Trade-offs

Agentless monitoring collects data over the network using existing protocols: SNMP, WMI, SSH exec, ICMP. No process runs on the target host. That sounds simpler, but the trade-offs accumulate fast at scale.

Agentless approaches depend on network access and protocol support from the target. For SNMP v2c, you get limited metric depth and community-string auth that most security teams flag immediately. SSH-based collection requires key management and adds latency on every poll cycle. WMI queries against Windows hosts are expensive – a single WMI query can consume 30-60ms on a loaded host, which compounds when you’re polling 200 servers every 30 seconds.

Agent-based monitoring puts a small process on the target host. That process reads metrics from local kernel interfaces – `/proc`, `sysfs`, Windows Performance Counters, network device MIBs compiled into the agent – and either exposes them for scraping or pushes them upstream. Local reads are fast and do not depend on network reachability to the target.

The real trade-off is operational: agents require deployment, configuration management, version control, and update pipelines. Agentless avoids that deployment overhead but limits metric granularity and increases collection latency. For any infrastructure where you need per-process metrics, disk I/O queue depth, or memory pressure data, agentless collection cannot match what a local agent reads directly from the kernel.

Pull Model: How Scraping Works

In a pull-based system, the monitoring server initiates each data collection cycle. Prometheus is the canonical implementation. The server holds a list of targets, sends an HTTP GET to each target’s metrics endpoint at a configured scrape interval, parses the response, and writes the samples to its time-series database.

The default scrape interval for most Prometheus deployments is 15 seconds, though 30 seconds is common on infrastructure with thousands of targets to reduce server load. Intervals below 10 seconds are rarely worth the cost unless you’re chasing sub-minute SLOs on latency-sensitive services.

Service discovery removes the need to maintain a static target list. Prometheus supports Kubernetes service discovery natively, reading pod and service annotations to build its target list dynamically. For bare-metal or VM fleets, file-based service discovery works well: a script or config management tool writes a JSON file listing targets, and Prometheus watches that file for changes without requiring a reload.

What the Scrape Actually Does

When Prometheus scrapes a target, it sends a plain HTTP GET. The exporter on the target reads current metric values from the OS or application, formats them as Prometheus exposition text, and returns them in the response body. The scrape duration, success status, and sample count become metrics themselves, stored as `scrape_duration_seconds` and `up`.

The `up` metric is often the first signal that an agent has stopped responding. If `up{job=”node”, instance=”10.0.1.15:9100″}` drops to 0, the node exporter on that host is unreachable. Whether the host is down, the exporter crashed, or a firewall rule changed is a separate question – but the absence of data triggers an alert before the absence of any other metric would.

Scrape Interval vs Resolution Trade-offs

Shorter scrape intervals give finer resolution but increase load on both the scraper and the target. At 15-second intervals across 500 targets, Prometheus issues roughly 2,000 HTTP requests per minute. Memory usage on the Prometheus server scales with the number of active time series, not just the number of targets. A single node with 10 standard exporters can expose 2,000 to 5,000 individual time series, so fleet size compounds quickly.

Longer intervals reduce load but widen the gap between when a problem starts and when metrics reflect it. A spike that lasts 20 seconds and resolves before the next 30-second scrape never appears in your data. For high-frequency signals like request error rates, push-based collection at the source often produces better resolution at lower infrastructure cost.

Push Model: When Push Makes More Sense Than Pull

Push-based agents send metrics to a central collector without waiting to be polled. Telegraf, StatsD, Elastic Beats, and the OpenTelemetry Protocol (OTLP) all operate this way. The agent decides when to flush, at what interval, and to which endpoint.

Push works better than pull in several specific scenarios. Short-lived jobs are the clearest case: a batch job that runs for 45 seconds and exits will likely fall between two scrape cycles and never appear in a pull-based system. Prometheus solves this with a Pushgateway, but that introduces state and staleness problems of its own. A push-based agent that flushes on exit captures the job’s metrics cleanly.

For architectures with dynamic ephemeral workloads – Lambda functions, Kubernetes Jobs, CI runners – push avoids the service discovery latency that pull-based systems require. See Microservices Monitoring Architecture and How to Monitor Distributed Systems for how these patterns apply at scale.

Telegraf, StatsD, and Beats

Telegraf is an agent written in Go that supports over 200 input plugins and dozens of output plugins. It reads from system interfaces, application APIs, and message queues, then pushes batched metrics to InfluxDB, Prometheus remote write, Kafka, or other backends. Its plugin model makes it adaptable, but the configuration surface is large. A misconfigured output plugin that silently drops metrics in an error condition is a real failure mode.

StatsD originated at Etsy and uses a UDP-based push protocol designed for application-level instrumentation. Applications send raw metric lines over UDP to a local StatsD daemon, which aggregates and forwards upstream. UDP means no delivery confirmation and no backpressure, which is acceptable for metrics but means you will lose data during collector restarts or network saturation.

Elastic Beats (Metricbeat, Filebeat, Packetbeat) push to Elasticsearch or Logstash. They work well when Elasticsearch is already the observability backend, but they add a heavyweight dependency if you’re running them solely for infrastructure metrics.

The Exporter Ecosystem: Node Exporter, Blackbox Exporter, Custom Exporters

In the Prometheus model, an exporter is a process that translates non-Prometheus metric sources into the Prometheus exposition format and serves them over HTTP. Exporters decouple the metric source from the collection mechanism. The Linux Monitoring Stack How-To covers the full stack setup in detail.

Node Exporter

Node Exporter is the standard for Linux host metrics. It reads from `/proc` and `/sys` and exposes CPU, memory, disk I/O, filesystem, and network interface metrics. A default installation exposes roughly 800 individual metrics per host. Most deployments enable a subset using the `–collector.disable-defaults` flag combined with explicit collector enables, which reduces the time series count and scrape payload size.

Node Exporter runs as an unprivileged process on port 9100. It requires no kernel modules or elevated permissions for most collectors. The `perf` and `ksmd` collectors are exceptions and require additional privileges, which most security policies restrict.

Blackbox Exporter

Blackbox Exporter performs synthetic probing from the Prometheus server’s perspective: HTTP, HTTPS, DNS, TCP, and ICMP probes. It does not run on the target. Instead, Prometheus scrapes the Blackbox Exporter with a target URL as a parameter, the exporter probes the target, and returns the result. This tests external reachability and TLS certificate validity, not internal host health. DNS Monitoring integrates naturally here: the Blackbox Exporter’s DNS module catches resolution failures and NXDOMAIN responses that agent-based monitoring on the host would never surface.

Custom Exporters

Custom exporters are necessary when you have internal applications or proprietary systems with no community exporter. Writing one in Go or Python using the official Prometheus client libraries takes a few hours for a simple case. The operational burden is the harder part: custom exporters need their own deployment pipeline, version management, and alerting on their own health.

A common mistake is writing a custom exporter that makes synchronous API calls inside the scrape handler. If the upstream API is slow or unavailable, the scrape times out, the `up` metric drops to 0, and the exporter appears down even though it’s the upstream that failed. Design exporters with internal caching: refresh metrics on a background goroutine and serve the cached result immediately on scrape.

Agent Overhead: Real CPU, Memory, and Network Cost

Every monitoring agent consumes resources on the host it monitors. The question is whether that cost is acceptable given the value of the data. For most infrastructure, it is – but the numbers matter.

Node Exporter on a modern Linux host uses 10-25MB of resident memory and under 0.1% CPU at a 15-second scrape interval on typical hardware. Telegraf with 10-15 input plugins runs higher: typically 50-80MB RSS and 0.3-0.8% CPU on an 8-core host during a collection cycle. These numbers scale with plugin count and collection frequency.

Network cost depends on metric volume and collection interval. A standard Node Exporter scrape response is roughly 50-100KB of uncompressed text. Prometheus supports gzip compression on scrapes, which reduces that to 8-15KB. At 15-second intervals, that’s about 40KB/minute per host – negligible on any modern network but worth accounting for across large fleets or across WAN links.

For deeper context on how these costs interact with host resource contention, see CPU Monitoring Explained and Disk I/O Monitoring Explained.

When Agent Overhead Becomes a Problem

Agent overhead becomes significant on hosts already under resource pressure. A host at 95% CPU utilization running a Telegraf agent with a 10-second collection interval adds context-switch pressure at the worst possible time. The agent does not cause the incident, but it can extend it.

For constrained environments – embedded systems, edge nodes with 512MB RAM, single-board compute – lightweight agents like the VictoriaMetrics `vmagent` or a stripped-down Node Exporter binary with minimal collectors reduce that overhead substantially. The trade-off is reduced metric coverage.

What Goes Wrong: Agent Failure Modes and the Data Gaps They Create

Agent failures are often silent. The agent process exits, no metrics flow, and depending on your alerting configuration, nothing fires for minutes or hours. These are the failure modes that matter most in production.

OOM Kill

A monitoring agent running on a host under memory pressure is a candidate for OOM kill. The kernel terminates the agent to recover memory, the agent leaves no trace beyond a kernel log entry, and your monitoring stack shows a clean `up=0` for that host. Since memory pressure also correlates with application incidents, you may lose agent coverage exactly when you most need it. See Memory Monitoring Explained for how to detect this condition before the OOM killer fires.

Configuration Drift

Agent configurations managed outside of a config management system drift over time. A scrape target gets a new port, a firewall rule changes, a TLS certificate rotates and the agent’s CA bundle is stale. Each of these silently stops data collection without crashing the agent. The agent is running and healthy by its own assessment; it simply cannot reach its data source or cannot connect to its upstream.

Clock Skew

Push-based agents timestamp their own metrics at collection time. If the host clock drifts more than a few seconds relative to the central collector, samples arrive out of order or are rejected as too old. VictoriaMetrics rejects samples more than 1 hour outside the current window by default. Prometheus remote write has similar bounds. An NTP failure on a single host can silently drop all metrics from that host without any obvious error in the agent log.

Agent Version Mismatch

Running different agent versions across a fleet creates inconsistent metric naming. Prometheus Node Exporter changed several metric names between major versions – `node_cpu` became `node_cpu_seconds_total` in v0.16. Mixed versions produce dashboards that show data for some hosts and not others, which appears as a host health problem rather than an agent version problem. Standardize agent versions via your deployment tooling and alert on version skew.

Cross-Platform Monitoring: Linux, Windows, Network Devices

Linux, Windows, and network devices require different agents and expose fundamentally different metric interfaces. A unified monitoring strategy has to account for that variation.

On Linux, Node Exporter and Telegraf read from `/proc` and `/sys`. These virtual filesystems expose kernel state directly — no system calls required for most metrics, just file reads. This makes Linux metric collection fast and low-overhead.

On Windows, the equivalent interfaces are Windows Management Instrumentation (WMI) and Windows Performance Counters. The `windows_exporter` (formerly `wmi_exporter`) reads from Performance Counters, which is faster and lower-overhead than WMI queries. WMI is available as a fallback but carries the latency cost mentioned earlier. Windows agents also deal with service management differently: `windows_exporter` runs as a Windows Service, not a systemd unit, which changes how you deploy, restart, and monitor the agent itself.

Network devices generally do not run third-party agents. The standard collection paths are SNMP polling (agentless, pull), streaming telemetry via gRPC (push, supported on modern Cisco, Juniper, and Arista platforms), and NETCONF/RESTCONF for configuration state. SNMP v2c remains dominant in practice despite its limitations. SNMPv3 with auth and encryption is required for any environment with network-layer security controls. For a detailed breakdown of these approaches, see How Monitoring Agents Work on Linux, Windows, and Network Devices and Network Monitoring Explained: Latency, Loss, and Dependency Mapping.

Challenges Specific to Each Platform

Linux challenges center on kernel version compatibility for newer collector types and on managing the large number of available Node Exporter collectors. Windows challenges center on WMI instability under load and on credential management for agentless collection fallbacks. Network device challenges center on SNMP polling scale: polling 500 interfaces at 60-second intervals with a single SNMP poller is achievable; polling 5,000 interfaces requires horizontal scaling of the polling layer or a move to streaming telemetry.

How BlueGrid.io Deploys and Manages Monitoring Agents for Clients

BlueGrid.io’s NOC operates 24/7 monitoring across client infrastructure spanning CDN providers, SaaS platforms, and cybersecurity vendors. Agent deployment and lifecycle management are a core part of that service, not a one-time setup step.

Agent deployments go through configuration management – Ansible and Salt are most common across the current client fleet – with agent versions pinned and tested before rollout. Each agent configuration is stored in version control. Changes to scrape targets, collection intervals, or enabled collectors go through the same review process as application code changes.

For new client onboards, the standard process starts with an inventory scan to identify what’s running, what platforms are present, and what firewall rules are in place. Agent selection follows from that inventory: Node Exporter for Linux hosts, `windows_exporter` for Windows, SNMP or streaming telemetry for network devices, and Blackbox Exporter probes for external service endpoints. Custom exporters are added for internal applications where community exporters don’t exist.

The NOC monitors agent health as a first-class concern. An `up=0` for any monitored host triggers an alert within the 1-hour incident response SLA. Agent restarts, version drift, and clock skew are tracked separately from host health metrics. A flapping agent that restarts every few minutes is often an early signal of memory pressure or a configuration problem on the host – it shows up in NOC queues before the application incident does.

Agent overhead is reviewed per client environment. For high-density container hosts running 50+ containers per node, collection configurations are tuned to reduce per-container metric cardinality. For edge nodes with constrained resources, lightweight agent builds replace standard ones. The goal is full visibility at the lowest sustainable resource cost.

If your team is managing agent deployments at scale and needs consistent coverage across a mixed fleet, BlueGrid.io’s NOC handles agent lifecycle management as part of its managed infrastructure monitoring service. Reach out to discuss how the BlueGrid.io team structures agent-based monitoring for environments like yours.