When Prometheus emerged from SoundCloud in 2012, it challenged the dominant paradigm of metrics collection. Rather than having applications push metrics to a central collector, Prometheus pulls metrics from applications. This seemingly simple architectural choice has profound implications for reliability, discovery, and operational simplicity.

Prometheus has since become the de facto standard for cloud-native metrics collection, with adoption spanning from startups to the largest technology companies. Its influence is so significant that the Cloud Native Computing Foundation (CNCF) graduated Prometheus as only its second project after Kubernetes.

Understanding Prometheus architecture isn't just about learning one tool—it's about understanding the design principles that govern modern observability systems.

The most distinctive architectural decision in Prometheus is its pull-based collection model. Traditional monitoring systems like Graphite use a push model—applications send metrics to the monitoring system. Prometheus inverts this: the server periodically scrapes metrics from targets.

Why Pull Works Better for Service Monitoring:

The pull model provides several operational advantages that may not be immediately obvious:

The Scrape Model:

At its core, Prometheus performs this loop for each target:

1. Discover targets via service discovery or static configuration
2. Scrape the /metrics endpoint via HTTP(S)
3. Parse the exposition format (typically OpenMetrics or Prometheus text format)
4. Add timestamps and labels to each metric sample
5. Store in the time-series database
6. Wait for the next scrape interval (typically 15-60 seconds)

This model means that your application only needs to expose an HTTP endpoint that returns current metric values. No client library configuration for endpoints, no network issues reaching collectors, no concerns about buffering metrics during outages.

Prometheus's architecture consists of several interconnected components, each with distinct responsibilities:

External Components:

• Alertmanager — Handles alert deduplication, grouping, routing, and notification delivery (email, Slack, PagerDuty, etc.)
• Pushgateway — Allows short-lived jobs to push metrics, which Prometheus then scrapes
• Exporters — Bridge applications that can't be directly instrumented (MySQL, Redis, Linux system metrics)
• Client Libraries — Instrument your applications (Go, Java, Python, Ruby, .NET, and more)

In dynamic environments like Kubernetes, service instances come and go constantly. Static configuration doesn't scale. Prometheus addresses this with powerful service discovery integrations that automatically find and track targets.

Service Discovery Mechanisms:

Prometheus supports numerous discovery mechanisms out of the box:

Relabeling: The Secret Weapon

Relabeling is Prometheus's powerful mechanism for transforming discovered metadata. It enables:

• Filtering targets — Keep or drop based on labels (e.g., only scrape pods with specific annotations)
• Modifying labels — Rename, copy, or transform labels
• Setting scrape parameters — Dynamic paths, ports, or schemes based on metadata
• Dropping labels — Remove high-cardinality or sensitive labels before storage

Relabeling happens in three phases:

1. relabel_configs — Before scraping (affects which targets are scraped)
2. metric_relabel_configs — After scraping (affects which metrics are stored)
3. alert_relabel_configs — Before sending to Alertmanager

Prometheus includes a custom time-series database designed specifically for metrics workloads. Understanding its architecture helps you tune performance and plan capacity.

The Data Model:

Every Prometheus time series is uniquely identified by a metric name and a set of key-value labels:

http_requests_total{method="GET", endpoint="/api/users", status="200"}

Internally, this becomes a series identifier that indexes a sequence of (timestamp, value) pairs. The combination of metric name and labels is fingerprinted into a unique series ID.

Storage Architecture:

Prometheus TSDB organizes data into blocks:

data/
├── 01BKGV7JBM69T2G1BGBGM6KB12/   # Block (2 hours of data)
│   ├── chunks/                    # Compressed sample data
│   │   └── 000001
│   ├── index                      # Label indexes for fast lookup
│   ├── meta.json                  # Block metadata
│   └── tombstones                 # Deleted series markers
├── 01BKGTZQ1SYQJTR4PB43C8PD98/   # Another block
├── 01BKGTZQ1XYQJTR4PB43C8PD99/   # Another block
├── lock
└── wal/                           # Write-ahead log (recent data)
    ├── 00000002
    └── checkpoint.00000001

Key Concepts:

• Write-Ahead Log (WAL): Incoming samples first go to the WAL for durability. If Prometheus crashes, the WAL replays on startup.
• Head Block: The most recent 2 hours of data, kept in memory for fast writes and queries.
• Compaction: Older blocks are periodically merged and compressed into larger blocks, improving query efficiency.
• Retention: Data older than the retention period (default: 15 days) is deleted.

Compression:

Prometheus uses specialized compression for time-series data:

• Timestamps: Delta-of-delta encoding (timestamps usually increment by fixed intervals)
• Values: XOR encoding (consecutive values are often similar)
• Achieved compression: Typically 1-2 bytes per sample (vs. 16 bytes uncompressed)

This aggressive compression enables Prometheus to store weeks of data for millions of time series on modest hardware.

Capacity Planning Formula:

disk_space = retention_seconds × samples_per_second × bytes_per_sample

Examples:
- 10000 series, 15s interval, 15 days retention, 1.5 bytes/sample
- samples/second = 10000 / 15 = 667
- disk = 15 × 24 × 3600 × 667 × 1.5 = ~1.3 GB

PromQL (Prometheus Query Language) is a powerful functional language for querying time series data. It's the foundation for dashboards, alerts, and ad-hoc analysis.

Query Types:

PromQL supports four data types:

• Instant vector: Set of time series, single sample per series at one timestamp
• Range vector: Set of time series, range of samples per series
• Scalar: Simple numeric floating point value
• String: Simple string value (rarely used)

Common Query Patterns:

Recording rules allow you to precompute frequently used or computationally expensive expressions and save them as new time series. This is critical for maintaining dashboard performance and reducing query load.

Why Recording Rules Matter:

• Query performance: Pre-aggregated data queries faster
• Dashboard consistency: Multiple dashboards use the same computed metrics
• Reduced load: Expensive queries run once per evaluation, not per dashboard load
• Simplified alerting: Alert on precomputed values instead of complex expressions

Naming Convention:

Prometheus recommends a consistent naming pattern for recording rules:

level:metric:operations

Where:

• level: The aggregation level (job, instance, namespace, etc.)
• metric: The metric name being aggregated
• operations: List of operations applied (rate5m, ratio, p99, etc.)

Examples:

• job:http_requests:rate5m — Requests per second by job
• instance:cpu_seconds:rate1m — CPU usage by instance
• namespace:pod_restarts:increase24h — Pod restarts by namespace over 24h

Prometheus was designed as a single-server system with local storage. For production reliability and scale, you need to understand HA patterns and federation.

High Availability Pattern:

The simplest HA approach is running two identical Prometheus instances scraping the same targets. Both instances have complete data, and you query either one (behind a load balancer).

Important considerations for HA:

• Both instances scrape independently—there's no data sharing
• Small timing differences mean data isn't perfectly identical
• Alertmanager handles alert deduplication from both instances
• For queries, use any instance (or both behind a load balancer)

Federation: Hierarchical Prometheus:

Federation allows a Prometheus server to scrape selected time series from another Prometheus server. This enables hierarchical architectures:

Federation Use Cases:

• Cross-datacenter visibility: Global Prometheus aggregates from regional instances
• Team isolation: Team Prometheus instances federated into platform view
• Long-term storage: Federate to an instance with longer retention

Thanos / Cortex / Mimir:

For enterprise scale, consider Prometheus-compatible systems that add:

• Global querying: Query across multiple Prometheus instances
• Long-term storage: Object storage (S3, GCS) for extended retention
• High availability: Writes replicated to multiple nodes
• Horizontal scaling: Ingest millions of series

Thanos, Cortex, and Mimir all maintain PromQL compatibility while adding these enterprise features.

Prometheus is powerful but not universal. Understanding its strengths and limitations helps you choose the right tool.

Key Limitations:

• Not for events: Prometheus samples at intervals; it can miss short-lived events. For individual request logging, use logs or traces.
• Not precisely accurate: The pull model samples at intervals, so brief spikes between scrapes may be missed.
• Single-node design: Local storage means scaling requires federation or Thanos/Cortex.
• Pull requires reachability: Prometheus must reach targets. For IoT devices behind NAT, consider pushing to Pushgateway or using remote_write.

Choosing Between Systems:

Prometheus's architecture choices—pull-based collection, service discovery, local TSDB, and the powerful PromQL language—make it the standard for cloud-native metrics. Understanding these components enables effective instrumentation, querying, and scaling.

What's Next:

With Prometheus architecture understood, the next page focuses on metric naming conventions. Proper naming is critical for query consistency, dashboard reuse, and maintainable observability infrastructure. We'll cover standard conventions, anti-patterns, and how to design metric names that scale with your organization.