Coordination Services - Learning Module

Loading content...

0/273

Consul: Service Mesh + KV

The Swiss Army Knife of Service Networking

When HashiCorp introduced Consul in 2014, they asked a provocative question: Why should service discovery, health checking, and coordination be separate tools?

ZooKeeper and etcd provide coordination primitives, but service discovery often requires additional layers — DNS servers, health check services, load balancers. Consul takes a different approach by providing an integrated platform that handles service discovery, health checking, key-value storage, and (with Consul Connect) a complete service mesh with mutual TLS and authorization.

This integration isn't just convenience — it's a fundamentally different architectural philosophy that trades simplicity for comprehensiveness.

What You Will Learn

By the end of this page, you will understand Consul's multi-datacenter architecture, its hybrid use of gossip (Serf) for membership and Raft for consensus, its first-class service discovery and health checking, the key-value store, and Consul Connect for service mesh. You'll also understand when Consul's integrated approach is the right choice.

What is Consul?

Consul is a multi-cloud service networking platform that provides:

Service Discovery: Register services and discover them via DNS or HTTP API
Health Checking: Monitor service health and route traffic accordingly
Key-Value Store: Distributed key-value storage for configuration and coordination
Service Mesh (Connect): Secure service-to-service communication with mTLS
Multi-Datacenter: First-class support for multi-datacenter and multi-cloud deployments

Unlike ZooKeeper and etcd, which are focused specifically on coordination, Consul aims to be a complete service networking solution.

Consul vs etcd vs ZooKeeper Feature Comparison
Feature	Consul	etcd	ZooKeeper
Service Discovery	Native, first-class	DIY with leases/watches	DIY with ephemeral znodes
Health Checking	Built-in (HTTP, TCP, Script, gRPC)	DIY with leases	DIY with sessions
Key-Value Store	Yes (coordination focus)	Yes (coordination focus)	Yes (znodes)
Service Mesh	Yes (Consul Connect)	No	No
Multi-Datacenter	Native WAN federation	Not designed for	Limited
DNS Interface	Built-in	No	No
UI	Yes	No (third-party)	No (third-party)
Consensus Protocol	Raft	Raft	ZAB
Membership Protocol	Gossip (Serf)	Raft only	Session-based

The Consul Architecture Philosophy

Consul's design centers on a key insight: service networking is a graph problem. Services need to discover each other, verify each other's health, authenticate each other, and authorize communication. By owning the entire graph, Consul can provide features that would require integrating multiple tools:

Intentions: Service-to-service authorization policies
Prepared Queries: Pre-defined, parameterized service lookups
Connect: Automatic mTLS between services
Configuration Entries: Centralized service configuration

When Integration Wins

Consul's integrated approach shines when you need the full stack: discovery, health checking, configuration, and secure communication. If you only need a key-value store for configuration, etcd is simpler. If you're already running Kubernetes, Kubernetes-native solutions may be preferred. Consul excels in multi-cloud and VM-based environments.

Consul Architecture

Consul uses a hybrid architecture with two distinct protocols working together:

Gossip Protocol (Serf): Used for membership, failure detection, and event broadcast
Raft Consensus: Used for the consistent key-value store and service catalog

This separation allows Consul to scale to large cluster sizes while maintaining strong consistency where needed.

Converting Mermaid diagram...

Server Nodes

Server nodes form the brain of a Consul datacenter:

Participate in Raft consensus for the consistent state
Store the service catalog, KV data, ACL tokens, and Intentions
Typically run 3 or 5 servers per datacenter for fault tolerance
One server is elected leader; all writes go through the leader
Can federate across datacenters via WAN gossip

Client Nodes (Agents)

Client agents run on every node that runs services:

Lightweight — no Raft participation
Register local services and run health checks
Cache service discovery results for performance
Forward state modifications to servers
Participate in LAN gossip for membership

Gossip Protocol Benefits

•Scalable Membership: Gossip scales to thousands of nodes without overloading servers
•Fast Failure Detection: Sub-second detection as nodes gossip constantly
•Low Overhead: Each node only talks to a few neighbors, not all nodes
•Partition Tolerance: Gossip continues working in partial partitions
•Event Broadcast: Custom events can be broadcast to all nodes efficiently

Two Gossip Pools

Consul maintains two separate gossip pools:

LAN Gossip: All agents in a datacenter (clients and servers) gossip with each other. Used for membership and local event broadcast. Fast convergence, optimized for low-latency networks.
WAN Gossip: Only servers participate, across datacenters. Used for cross-datacenter health checking and routing. Tuned for higher latency, lower bandwidth.

Why Not Just Use Raft Everywhere?

Raft requires all writes to go through a single leader. With thousands of nodes joining, leaving, and failing, Raft would become a bottleneck. Gossip distributes the membership load: each node only talks to a few neighbors, and information propagates exponentially. This is why Consul can scale to 10,000+ nodes per datacenter.

Service Discovery and Health Checking

Service discovery is Consul's core strength. Unlike etcd and ZooKeeper where you build service discovery on top of primitives, Consul provides it as a first-class feature with built-in health checking, DNS interface, and prepared queries.

Service Registration

Services register with their local Consul agent (not directly with servers). The agent handles:

Local health checks for the service
Reporting health status to the cluster
Removing the service if health checks fail
Re-registering if the agent restarts

service-registration.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
  "service": {
    "name": "web-api",
    "id": "web-api-10.0.0.1:8080",
    "port": 8080,
    "address": "10.0.0.1",
    "tags": ["production", "v2"],
    "meta": {
      "version": "2.0.0",
      "team": "backend"
    },
    "check": {
      "http": "http://10.0.0.1:8080/health",
      "interval": "10s",
      "timeout": "5s",
      "deregister_critical_service_after": "1m"
    },
    "weights": {
      "passing": 10,
      "warning": 1
    }
  }
}

Health Check Types

Consul supports multiple health check types:

HTTP: GET request to endpoint, check status code
TCP: Attempt to connect to port
gRPC: gRPC health check protocol
Script: Run a local script, check exit code
TTL: Service must heartbeat within interval
Docker: Run command inside a Docker container
Alias: Inherit health from another service

Health checks run locally on the agent, minimizing network overhead and enabling sub-second detection.

service-discovery.sh
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# DNS-based discovery (simple, works with any application)
$ dig @127.0.0.1 -p 8600 web-api.service.consul
;; ANSWER SECTION:
web-api.service.consul. 0 IN A 10.0.0.1
web-api.service.consul. 0 IN A 10.0.0.2
web-api.service.consul. 0 IN A 10.0.0.3
 
# SRV records include port information
$ dig @127.0.0.1 -p 8600 web-api.service.consul SRV
;; ANSWER SECTION:
web-api.service.consul. 0 IN SRV 1 1 8080 10.0.0.1.node.dc1.consul.
web-api.service.consul. 0 IN SRV 1 1 8080 10.0.0.2.node.dc1.consul.
 
# Filter by tag
$ dig @127.0.0.1 -p 8600 v2.web-api.service.consul
 
# Cross-datacenter query
$ dig @127.0.0.1 -p 8600 web-api.service.dc2.consul
 
# HTTP API - more details including health
$ curl http://localhost:8500/v1/health/service/web-api?passing=true
[
  {
    "Node": {"Node": "node1", "Address": "10.0.0.1"},
    "Service": {"Service": "web-api", "Port": 8080},
    "Checks": [
      {"Status": "passing", "Output": "HTTP GET http://...: 200 OK"}
    ]
  }
]
 
# Catalog query (no health filtering, just registered services)
$ curl http://localhost:8500/v1/catalog/service/web-api

DNS Integration Superpower

Consul's DNS interface is uniquely powerful: any application that can resolve DNS can use Consul for service discovery without code changes. Point your DNS resolver at Consul's port 8600 (or use dnsmasq to forward .consul queries), and 'web-api.service.consul' just works.

Prepared Queries — Advanced Discovery

•Failover: Automatically query another datacenter if local service is unavailable
•Nearest: Route to geographically nearest healthy instances
•Tag Filtering: Pre-defined queries with tag filtering
•OnlyPassing: Only return instances with passing health checks
•DNS Integration: Access prepared queries via DNS (query-name.query.consul)

The Key-Value Store

Consul's key-value store provides the same coordination capabilities as etcd and ZooKeeper: configuration storage, distributed locks, leader election, and semaphores. While not as feature-rich as dedicated coordination services, it's sufficient for many use cases and benefits from Consul's operational simplicity.

kv-operations.sh
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Put a key
$ consul kv put config/database/connection-string "postgres://..."
Success! Data written to: config/database/connection-string
 
# Get a key
$ consul kv get config/database/connection-string
postgres://...
 
# Get with metadata
$ consul kv get -detailed config/database/connection-string
CreateIndex      42
Flags            0
Key              config/database/connection-string
LockIndex        0
ModifyIndex      42
Session          (none)
Value            postgres://...
 
# List keys with prefix
$ consul kv get -recurse config/
config/database/connection-string:postgres://...
config/database/pool-size:10
config/features/dark-mode:true
 
# Delete
$ consul kv delete config/old-key
 
# Delete prefix (recursive)
$ consul kv delete -recurse config/deprecated/
 
# Export/Import for backup
$ consul kv export config/ > config-backup.json
$ consul kv import @config-backup.json

Check-And-Set (CAS)

Consul supports optimistic locking via Check-And-Set operations. The ModifyIndex field serves as a version number:

cas-example.sh
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Get current value and ModifyIndex
$ curl -s http://localhost:8500/v1/kv/config/counter | jq
[{
  "Key": "config/counter",
  "Value": "MTAw",  # base64 of "100"
  "ModifyIndex": 42
}]
 
# CAS: only update if ModifyIndex still matches
$ curl -X PUT -d "101"   "http://localhost:8500/v1/kv/config/counter?cas=42"
true  # Success - value was updated
 
# If someone else modified it first:
$ curl -X PUT -d "101"   "http://localhost:8500/v1/kv/config/counter?cas=42"
false  # Failed - ModifyIndex changed, retry needed

Sessions and Locks

Consul provides a session abstraction for implementing ephemeral keys and distributed locks. Sessions have configurable behaviors on failure:

Release: Keys locked by the session are unlocked (default)
Delete: Keys locked by the session are deleted

Sessions are tied to a node and have TTL-based expiration.

session-locks.sh
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# Create a session with 30s TTL
$ curl -X PUT -d '{"Name": "my-session", "TTL": "30s", "Behavior": "release"}'     http://localhost:8500/v1/session/create
{"ID": "adf4238a-882b-9ddc-4a9d-5b6758e4159e"}
 
# Acquire a lock using the session
$ curl -X PUT -d "my-lock-value"     "http://localhost:8500/v1/kv/locks/myresource?acquire=adf4238a-882b-9ddc-4a9d-5b6758e4159e"
true  # Lock acquired
 
# Check who holds the lock
$ consul kv get -detailed locks/myresource
...
Session          adf4238a-882b-9ddc-4a9d-5b6758e4159e
...
 
# Another session trying to acquire will fail
$ curl -X PUT -d "other-value"     "http://localhost:8500/v1/kv/locks/myresource?acquire=other-session-id"
false  # Already locked
 
# Release the lock
$ curl -X PUT     "http://localhost:8500/v1/kv/locks/myresource?release=adf4238a-882b-9ddc-4a9d-5b6758e4159e"
true
 
# Renew session before TTL expires (keep-alive)
$ curl -X PUT     "http://localhost:8500/v1/session/renew/adf4238a-882b-9ddc-4a9d-5b6758e4159e"

Consul KV vs etcd/ZooKeeper

Consul's KV is simpler than etcd's transaction system and ZooKeeper's watch model. It lacks multi-key transactions and has less sophisticated watch semantics. For complex coordination, dedicated stores may be better. Consul KV excels when you need simple configuration storage alongside Consul's service discovery.

Consul Connect — Service Mesh

Consul Connect is Consul's service mesh feature, providing:

Mutual TLS (mTLS): Automatic encryption and authentication between services
Service Identity: Each service gets a cryptographic identity (SPIFFE-based)
Intentions: Service-to-service authorization policies
Traffic Management: Splitting, routing, and load balancing

This is where Consul differentiates most from etcd and ZooKeeper — they're coordination services, while Consul is a complete service networking solution.

How Connect Works

Connect can operate in two modes:

Sidecar Proxy: Each service gets a proxy (typically Envoy) that handles mTLS and traffic. This is transparent to the application.
Native Integration: Applications use Consul's Connect SDK to handle mTLS directly. Lower overhead but requires code changes.

Converting Mermaid diagram...

intentions.hcl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# Service mesh intentions - who can talk to whom
 
# Allow web-frontend to access web-api
Kind = "service-intentions"
Name = "web-api"
Sources = [
  {
    Name   = "web-frontend"
    Action = "allow"
  },
  {
    Name   = "monitoring"
    Action = "allow"
  },
  # Deny all other services (default deny)
  {
    Name   = "*"
    Action = "deny"
  }
]
 
# More complex: L7 routing based on HTTP paths
Kind = "service-intentions"
Name = "payment-api"
Sources = [
  {
    Name = "order-service"
    Permissions = [
      {
        Action = "allow"
        HTTP {
          PathPrefix = "/v2/"
          Methods    = ["GET", "POST"]
        }
      },
      {
        Action = "deny"
        HTTP {
          PathPrefix = "/admin/"
        }
      }
    ]
  }
]

Connect Features

•Zero Trust Networking: Every connection is authenticated and authorized, no implicit trust based on network location
•Automatic Certificate Rotation: Consul's CA issues short-lived certificates, automatically rotated before expiration
•SPIFFE Identity: Services get SPIFFE IDs (URIs like spiffe://cluster.local/ns/default/dc/dc1/svc/web-api)
•Traffic Splitting: Route percentages of traffic to different versions for canary deployments
•Service Resolvers: Control which instances resolve for service discovery queries
•Service Routers: Path-based and header-based routing to different service subsets

Consul vs Istio vs Linkerd

All are service mesh solutions, but Consul is uniquely designed for multi-cloud and VM environments. Istio and Linkerd are Kubernetes-focused. Consul's hybrid approach (VMs + Kubernetes + cloud) and its integration with HashiCorp tools (Vault, Terraform, Nomad) make it attractive for heterogeneous environments.

Multi-Datacenter and Federation

One of Consul's most distinctive features is first-class multi-datacenter support. While etcd and ZooKeeper clusters are typically confined to a single datacenter, Consul is designed from the ground up for global deployments.

How Federation Works

Each datacenter has its own independent Consul cluster (3-5 servers)
Servers in each datacenter join a global WAN gossip pool
WAN gossip enables cross-datacenter health checking and routing
Each datacenter maintains its own Raft consensus (no cross-DC Raft)
Services can discover services in remote datacenters
Requests are automatically routed to the appropriate datacenter

multi-dc-discovery.sh
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Query service in a specific datacenter
$ dig @127.0.0.1 -p 8600 web-api.service.us-west-2.consul
$ dig @127.0.0.1 -p 8600 web-api.service.eu-central-1.consul
 
# Prepared query with failover
$ curl -X POST -d '{
  "Name": "web-api-failover",
  "Service": {
    "Service": "web-api",
    "OnlyPassing": true,
    "Failover": {
      "NearestN": 2,
      "Datacenters": ["us-west-2", "us-east-1", "eu-central-1"]
    }
  }
}' http://localhost:8500/v1/query
 
# Query by prepared query name
$ dig @127.0.0.1 -p 8600 web-api-failover.query.consul
 
# Lists all datacenters
$ consul catalog datacenters
dc1
dc2
us-west-2
eu-central-1
 
# KV data is NOT automatically replicated across datacenters
# For cross-DC KV, manually replicate or use consul-replicate tool

KV Is Per-Datacenter

Unlike services, Consul's KV store is NOT automatically replicated across datacenters. Each datacenter has independent KV data. For cross-DC configuration, use tools like consul-replicate or store configuration in each datacenter separately. This is intentional—cross-DC Raft would introduce unacceptable latency.

Cross-Datacenter Behavior
Feature	Cross-DC Behavior
Service Discovery	Yes - can query services in any federated DC
Health Checks	Yes - WAN gossip propagates health status
Prepared Queries	Yes - failover and nearest routing work cross-DC
KV Store	No automatic replication - per-DC data
ACL Tokens	Replicated from primary DC (ACL replication)
Intentions	Replicated across DCs for mesh connectivity
Raft Consensus	Independent per-DC - no cross-DC Raft

Mesh Gateways

For Connect traffic across datacenters, Consul uses mesh gateways:

Dedicated proxy that routes mTLS traffic between datacenters
Terminates and re-originates mTLS at datacenter boundary
Enables routing even when services can't directly reach each other
Essential for multi-cloud where networks don't interconnect

Running Consul in Production

Consul's comprehensive feature set comes with operational considerations. Here's what you need to know for production deployments.

Production Deployment Checklist

•3 or 5 servers per datacenter — Never run 2 or 4; Raft requires odd numbers for clear quorum
•Dedicated server machines — Don't co-locate Consul servers with application workloads
•SSD storage — Consul (like all Raft-based systems) is latency-sensitive
•Enable ACLs — Default deny policy; all access requires explicit tokens
•Enable TLS — encrypt gossip and RPC traffic; especially critical for WAN
•Snapshot storage — Regular backups via consul snapshot for disaster recovery
•Monitor disk usage — Consul's data directory grows with service catalog and snapshots

consul-server.hcl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
# Production server configuration example
datacenter = "us-west-2"
data_dir   = "/opt/consul/data"
log_level  = "INFO"
 
server           = true
bootstrap_expect = 3
 
# Bind addresses
bind_addr   = "10.0.0.1"
client_addr = "0.0.0.0"
 
# TLS configuration
tls {
  defaults {
    verify_incoming = true
    verify_outgoing = true
    ca_file         = "/etc/consul.d/certs/consul-ca.pem"
    cert_file       = "/etc/consul.d/certs/consul-server.pem"
    key_file        = "/etc/consul.d/certs/consul-server-key.pem"
  }
}
 
# Gossip encryption
encrypt = "base64-encoded-32-byte-key"
 
# ACL configuration
acl {
  enabled                  = true
  default_policy           = "deny"
  enable_token_persistence = true
}
 
# Performance tuning
performance {
  raft_multiplier = 1  # Lower = faster, but more sensitive to slowness
}
 
# Autopilot for automatic server management
autopilot {
  cleanup_dead_servers      = true
  last_contact_threshold    = "200ms"
  max_trailing_logs         = 250
  server_stabilization_time = "10s"
}

Autopilot for Cluster Management

Consul's Autopilot feature automatically manages server lifecycle: dead server cleanup, server health monitoring, and stable leader election. It's especially useful for auto-scaling environments where servers come and go.

Common Operational Issues

•Anti-entropy sync storms — Large service catalogs can cause periodic CPU spikes; tune sync intervals
•DNS performance — For high-volume DNS, run local caching resolvers (dnsmasq/unbound)
•ACL token management — Use short-lived tokens; integrate with Vault for dynamic secrets
•Gossip encryption rotation — Rotate gossip keys without downtime using the dual-key period
•Certificate rotation — With Connect, ensure sufficient CA certificate validity

Summary: Consul Essentials

We've covered Consul's integrated approach to service networking. Let's consolidate the essential knowledge:

Key Takeaways

•Consul is a service networking platform, not just a coordination service — it combines discovery, health checking, KV, and service mesh.
•Hybrid consensus architecture — Gossip (Serf) for scalable membership, Raft for consistent state storage.
•First-class service discovery with DNS interface, health checking, and prepared queries for failover.
•Consul Connect provides automatic mTLS and intentions-based authorization for zero-trust networking.
•Native multi-datacenter with WAN gossip, cross-DC discovery, and mesh gateways.
•KV store provides coordination primitives but is simpler than dedicated stores like etcd.
•Operational complexity is higher than single-purpose tools but lower than integrating multiple systems.
•Best for heterogeneous environments — VMs, containers, multi-cloud, where a unified platform adds value.

When to Choose Consul

•Multi-cloud or hybrid deployments
•VM-based infrastructure
•Need integrated service mesh
•Multi-datacenter requirements
•Using HashiCorp ecosystem
•Want DNS-based discovery

When Other Tools May Be Better

•Kubernetes-only (use native K8s + Istio/Linkerd)
•Only need KV store (use etcd)
•Complex coordination patterns (use ZooKeeper)
•Minimal operational overhead priority
•Single datacenter, simple needs

What's Next:

In the next page, we'll systematically compare ZooKeeper, etcd, and Consul to help you choose the right coordination service for your specific needs. You'll learn the key criteria for making this architectural decision.

Page Complete

You now understand Consul's integrated approach to service networking, its hybrid architecture with gossip and Raft, and when its comprehensive feature set is the right choice. This foundation prepares you to make informed comparisons between coordination services.

Consul: Service Mesh + KV

The Swiss Army Knife of Service Networking

When HashiCorp introduced Consul in 2014, they asked a provocative question: Why should service discovery, health checking, and coordination be separate tools?

This integration isn't just convenience — it's a fundamentally different architectural philosophy that trades simplicity for comprehensiveness.

What You Will Learn

What is Consul?

Consul is a multi-cloud service networking platform that provides:

Service Discovery: Register services and discover them via DNS or HTTP API
Health Checking: Monitor service health and route traffic accordingly
Key-Value Store: Distributed key-value storage for configuration and coordination
Service Mesh (Connect): Secure service-to-service communication with mTLS
Multi-Datacenter: First-class support for multi-datacenter and multi-cloud deployments

Unlike ZooKeeper and etcd, which are focused specifically on coordination, Consul aims to be a complete service networking solution.

Consul vs etcd vs ZooKeeper Feature Comparison
Feature	Consul	etcd	ZooKeeper
Service Discovery	Native, first-class	DIY with leases/watches	DIY with ephemeral znodes
Health Checking	Built-in (HTTP, TCP, Script, gRPC)	DIY with leases	DIY with sessions
Key-Value Store	Yes (coordination focus)	Yes (coordination focus)	Yes (znodes)
Service Mesh	Yes (Consul Connect)	No	No
Multi-Datacenter	Native WAN federation	Not designed for	Limited
DNS Interface	Built-in	No	No
UI	Yes	No (third-party)	No (third-party)
Consensus Protocol	Raft	Raft	ZAB
Membership Protocol	Gossip (Serf)	Raft only	Session-based

The Consul Architecture Philosophy

Intentions: Service-to-service authorization policies
Prepared Queries: Pre-defined, parameterized service lookups
Connect: Automatic mTLS between services
Configuration Entries: Centralized service configuration

When Integration Wins

Consul Architecture

Consul uses a hybrid architecture with two distinct protocols working together:

Gossip Protocol (Serf): Used for membership, failure detection, and event broadcast
Raft Consensus: Used for the consistent key-value store and service catalog

This separation allows Consul to scale to large cluster sizes while maintaining strong consistency where needed.

Converting Mermaid diagram...

Server Nodes

Server nodes form the brain of a Consul datacenter:

Participate in Raft consensus for the consistent state
Store the service catalog, KV data, ACL tokens, and Intentions
Typically run 3 or 5 servers per datacenter for fault tolerance
One server is elected leader; all writes go through the leader
Can federate across datacenters via WAN gossip

Client Nodes (Agents)

Client agents run on every node that runs services:

Lightweight — no Raft participation
Register local services and run health checks
Cache service discovery results for performance
Forward state modifications to servers
Participate in LAN gossip for membership

Gossip Protocol Benefits

•Scalable Membership: Gossip scales to thousands of nodes without overloading servers
•Fast Failure Detection: Sub-second detection as nodes gossip constantly
•Low Overhead: Each node only talks to a few neighbors, not all nodes
•Partition Tolerance: Gossip continues working in partial partitions
•Event Broadcast: Custom events can be broadcast to all nodes efficiently

Two Gossip Pools

Consul maintains two separate gossip pools:

LAN Gossip: All agents in a datacenter (clients and servers) gossip with each other. Used for membership and local event broadcast. Fast convergence, optimized for low-latency networks.
WAN Gossip: Only servers participate, across datacenters. Used for cross-datacenter health checking and routing. Tuned for higher latency, lower bandwidth.

Why Not Just Use Raft Everywhere?

Service Discovery and Health Checking

Service Registration

Services register with their local Consul agent (not directly with servers). The agent handles:

Local health checks for the service
Reporting health status to the cluster
Removing the service if health checks fail
Re-registering if the agent restarts

service-registration.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
  "service": {
    "name": "web-api",
    "id": "web-api-10.0.0.1:8080",
    "port": 8080,
    "address": "10.0.0.1",
    "tags": ["production", "v2"],
    "meta": {
      "version": "2.0.0",
      "team": "backend"
    },
    "check": {
      "http": "http://10.0.0.1:8080/health",
      "interval": "10s",
      "timeout": "5s",
      "deregister_critical_service_after": "1m"
    },
    "weights": {
      "passing": 10,
      "warning": 1
    }
  }
}

Health Check Types

Consul supports multiple health check types:

HTTP: GET request to endpoint, check status code
TCP: Attempt to connect to port
gRPC: gRPC health check protocol
Script: Run a local script, check exit code
TTL: Service must heartbeat within interval
Docker: Run command inside a Docker container
Alias: Inherit health from another service

Health checks run locally on the agent, minimizing network overhead and enabling sub-second detection.

service-discovery.sh
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# DNS-based discovery (simple, works with any application)
$ dig @127.0.0.1 -p 8600 web-api.service.consul
;; ANSWER SECTION:
web-api.service.consul. 0 IN A 10.0.0.1
web-api.service.consul. 0 IN A 10.0.0.2
web-api.service.consul. 0 IN A 10.0.0.3
 
# SRV records include port information
$ dig @127.0.0.1 -p 8600 web-api.service.consul SRV
;; ANSWER SECTION:
web-api.service.consul. 0 IN SRV 1 1 8080 10.0.0.1.node.dc1.consul.
web-api.service.consul. 0 IN SRV 1 1 8080 10.0.0.2.node.dc1.consul.
 
# Filter by tag
$ dig @127.0.0.1 -p 8600 v2.web-api.service.consul
 
# Cross-datacenter query
$ dig @127.0.0.1 -p 8600 web-api.service.dc2.consul
 
# HTTP API - more details including health
$ curl http://localhost:8500/v1/health/service/web-api?passing=true
[
  {
    "Node": {"Node": "node1", "Address": "10.0.0.1"},
    "Service": {"Service": "web-api", "Port": 8080},
    "Checks": [
      {"Status": "passing", "Output": "HTTP GET http://...: 200 OK"}
    ]
  }
]
 
# Catalog query (no health filtering, just registered services)
$ curl http://localhost:8500/v1/catalog/service/web-api

DNS Integration Superpower

Prepared Queries — Advanced Discovery

•Failover: Automatically query another datacenter if local service is unavailable
•Nearest: Route to geographically nearest healthy instances
•Tag Filtering: Pre-defined queries with tag filtering
•OnlyPassing: Only return instances with passing health checks
•DNS Integration: Access prepared queries via DNS (query-name.query.consul)

The Key-Value Store

kv-operations.sh
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Put a key
$ consul kv put config/database/connection-string "postgres://..."
Success! Data written to: config/database/connection-string
 
# Get a key
$ consul kv get config/database/connection-string
postgres://...
 
# Get with metadata
$ consul kv get -detailed config/database/connection-string
CreateIndex      42
Flags            0
Key              config/database/connection-string
LockIndex        0
ModifyIndex      42
Session          (none)
Value            postgres://...
 
# List keys with prefix
$ consul kv get -recurse config/
config/database/connection-string:postgres://...
config/database/pool-size:10
config/features/dark-mode:true
 
# Delete
$ consul kv delete config/old-key
 
# Delete prefix (recursive)
$ consul kv delete -recurse config/deprecated/
 
# Export/Import for backup
$ consul kv export config/ > config-backup.json
$ consul kv import @config-backup.json

Check-And-Set (CAS)

Consul supports optimistic locking via Check-And-Set operations. The ModifyIndex field serves as a version number:

cas-example.sh
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Get current value and ModifyIndex
$ curl -s http://localhost:8500/v1/kv/config/counter | jq
[{
  "Key": "config/counter",
  "Value": "MTAw",  # base64 of "100"
  "ModifyIndex": 42
}]
 
# CAS: only update if ModifyIndex still matches
$ curl -X PUT -d "101"   "http://localhost:8500/v1/kv/config/counter?cas=42"
true  # Success - value was updated
 
# If someone else modified it first:
$ curl -X PUT -d "101"   "http://localhost:8500/v1/kv/config/counter?cas=42"
false  # Failed - ModifyIndex changed, retry needed

Sessions and Locks

Consul provides a session abstraction for implementing ephemeral keys and distributed locks. Sessions have configurable behaviors on failure:

Release: Keys locked by the session are unlocked (default)
Delete: Keys locked by the session are deleted

Sessions are tied to a node and have TTL-based expiration.

session-locks.sh
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
# Create a session with 30s TTL
$ curl -X PUT -d '{"Name": "my-session", "TTL": "30s", "Behavior": "release"}'     http://localhost:8500/v1/session/create
{"ID": "adf4238a-882b-9ddc-4a9d-5b6758e4159e"}
 
# Acquire a lock using the session
$ curl -X PUT -d "my-lock-value"     "http://localhost:8500/v1/kv/locks/myresource?acquire=adf4238a-882b-9ddc-4a9d-5b6758e4159e"
true  # Lock acquired
 
# Check who holds the lock
$ consul kv get -detailed locks/myresource
...
Session          adf4238a-882b-9ddc-4a9d-5b6758e4159e
...
 
# Another session trying to acquire will fail
$ curl -X PUT -d "other-value"     "http://localhost:8500/v1/kv/locks/myresource?acquire=other-session-id"
false  # Already locked
 
# Release the lock
$ curl -X PUT     "http://localhost:8500/v1/kv/locks/myresource?release=adf4238a-882b-9ddc-4a9d-5b6758e4159e"
true
 
# Renew session before TTL expires (keep-alive)
$ curl -X PUT     "http://localhost:8500/v1/session/renew/adf4238a-882b-9ddc-4a9d-5b6758e4159e"

Consul KV vs etcd/ZooKeeper

Consul Connect — Service Mesh

Consul Connect is Consul's service mesh feature, providing:

Mutual TLS (mTLS): Automatic encryption and authentication between services
Service Identity: Each service gets a cryptographic identity (SPIFFE-based)
Intentions: Service-to-service authorization policies
Traffic Management: Splitting, routing, and load balancing

This is where Consul differentiates most from etcd and ZooKeeper — they're coordination services, while Consul is a complete service networking solution.

How Connect Works

Connect can operate in two modes:

Sidecar Proxy: Each service gets a proxy (typically Envoy) that handles mTLS and traffic. This is transparent to the application.
Native Integration: Applications use Consul's Connect SDK to handle mTLS directly. Lower overhead but requires code changes.

Converting Mermaid diagram...

intentions.hcl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# Service mesh intentions - who can talk to whom
 
# Allow web-frontend to access web-api
Kind = "service-intentions"
Name = "web-api"
Sources = [
  {
    Name   = "web-frontend"
    Action = "allow"
  },
  {
    Name   = "monitoring"
    Action = "allow"
  },
  # Deny all other services (default deny)
  {
    Name   = "*"
    Action = "deny"
  }
]
 
# More complex: L7 routing based on HTTP paths
Kind = "service-intentions"
Name = "payment-api"
Sources = [
  {
    Name = "order-service"
    Permissions = [
      {
        Action = "allow"
        HTTP {
          PathPrefix = "/v2/"
          Methods    = ["GET", "POST"]
        }
      },
      {
        Action = "deny"
        HTTP {
          PathPrefix = "/admin/"
        }
      }
    ]
  }
]

Connect Features

•Zero Trust Networking: Every connection is authenticated and authorized, no implicit trust based on network location
•Automatic Certificate Rotation: Consul's CA issues short-lived certificates, automatically rotated before expiration
•SPIFFE Identity: Services get SPIFFE IDs (URIs like spiffe://cluster.local/ns/default/dc/dc1/svc/web-api)
•Traffic Splitting: Route percentages of traffic to different versions for canary deployments
•Service Resolvers: Control which instances resolve for service discovery queries
•Service Routers: Path-based and header-based routing to different service subsets

Consul vs Istio vs Linkerd

Multi-Datacenter and Federation

How Federation Works

Each datacenter has its own independent Consul cluster (3-5 servers)
Servers in each datacenter join a global WAN gossip pool
WAN gossip enables cross-datacenter health checking and routing
Each datacenter maintains its own Raft consensus (no cross-DC Raft)
Services can discover services in remote datacenters
Requests are automatically routed to the appropriate datacenter

multi-dc-discovery.sh
Shell
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Query service in a specific datacenter
$ dig @127.0.0.1 -p 8600 web-api.service.us-west-2.consul
$ dig @127.0.0.1 -p 8600 web-api.service.eu-central-1.consul
 
# Prepared query with failover
$ curl -X POST -d '{
  "Name": "web-api-failover",
  "Service": {
    "Service": "web-api",
    "OnlyPassing": true,
    "Failover": {
      "NearestN": 2,
      "Datacenters": ["us-west-2", "us-east-1", "eu-central-1"]
    }
  }
}' http://localhost:8500/v1/query
 
# Query by prepared query name
$ dig @127.0.0.1 -p 8600 web-api-failover.query.consul
 
# Lists all datacenters
$ consul catalog datacenters
dc1
dc2
us-west-2
eu-central-1
 
# KV data is NOT automatically replicated across datacenters
# For cross-DC KV, manually replicate or use consul-replicate tool

KV Is Per-Datacenter

Cross-Datacenter Behavior
Feature	Cross-DC Behavior
Service Discovery	Yes - can query services in any federated DC
Health Checks	Yes - WAN gossip propagates health status
Prepared Queries	Yes - failover and nearest routing work cross-DC
KV Store	No automatic replication - per-DC data
ACL Tokens	Replicated from primary DC (ACL replication)
Intentions	Replicated across DCs for mesh connectivity
Raft Consensus	Independent per-DC - no cross-DC Raft

Mesh Gateways

For Connect traffic across datacenters, Consul uses mesh gateways:

Dedicated proxy that routes mTLS traffic between datacenters
Terminates and re-originates mTLS at datacenter boundary
Enables routing even when services can't directly reach each other
Essential for multi-cloud where networks don't interconnect

Running Consul in Production

Consul's comprehensive feature set comes with operational considerations. Here's what you need to know for production deployments.

Production Deployment Checklist

•3 or 5 servers per datacenter — Never run 2 or 4; Raft requires odd numbers for clear quorum
•Dedicated server machines — Don't co-locate Consul servers with application workloads
•SSD storage — Consul (like all Raft-based systems) is latency-sensitive
•Enable ACLs — Default deny policy; all access requires explicit tokens
•Enable TLS — encrypt gossip and RPC traffic; especially critical for WAN
•Snapshot storage — Regular backups via consul snapshot for disaster recovery
•Monitor disk usage — Consul's data directory grows with service catalog and snapshots

consul-server.hcl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
# Production server configuration example
datacenter = "us-west-2"
data_dir   = "/opt/consul/data"
log_level  = "INFO"
 
server           = true
bootstrap_expect = 3
 
# Bind addresses
bind_addr   = "10.0.0.1"
client_addr = "0.0.0.0"
 
# TLS configuration
tls {
  defaults {
    verify_incoming = true
    verify_outgoing = true
    ca_file         = "/etc/consul.d/certs/consul-ca.pem"
    cert_file       = "/etc/consul.d/certs/consul-server.pem"
    key_file        = "/etc/consul.d/certs/consul-server-key.pem"
  }
}
 
# Gossip encryption
encrypt = "base64-encoded-32-byte-key"
 
# ACL configuration
acl {
  enabled                  = true
  default_policy           = "deny"
  enable_token_persistence = true
}
 
# Performance tuning
performance {
  raft_multiplier = 1  # Lower = faster, but more sensitive to slowness
}
 
# Autopilot for automatic server management
autopilot {
  cleanup_dead_servers      = true
  last_contact_threshold    = "200ms"
  max_trailing_logs         = 250
  server_stabilization_time = "10s"
}

Autopilot for Cluster Management

Common Operational Issues

•Anti-entropy sync storms — Large service catalogs can cause periodic CPU spikes; tune sync intervals
•DNS performance — For high-volume DNS, run local caching resolvers (dnsmasq/unbound)
•ACL token management — Use short-lived tokens; integrate with Vault for dynamic secrets
•Gossip encryption rotation — Rotate gossip keys without downtime using the dual-key period
•Certificate rotation — With Connect, ensure sufficient CA certificate validity

Summary: Consul Essentials

We've covered Consul's integrated approach to service networking. Let's consolidate the essential knowledge:

Key Takeaways

•Consul is a service networking platform, not just a coordination service — it combines discovery, health checking, KV, and service mesh.
•Hybrid consensus architecture — Gossip (Serf) for scalable membership, Raft for consistent state storage.
•First-class service discovery with DNS interface, health checking, and prepared queries for failover.
•Consul Connect provides automatic mTLS and intentions-based authorization for zero-trust networking.
•Native multi-datacenter with WAN gossip, cross-DC discovery, and mesh gateways.
•KV store provides coordination primitives but is simpler than dedicated stores like etcd.
•Operational complexity is higher than single-purpose tools but lower than integrating multiple systems.
•Best for heterogeneous environments — VMs, containers, multi-cloud, where a unified platform adds value.

When to Choose Consul

•Multi-cloud or hybrid deployments
•VM-based infrastructure
•Need integrated service mesh
•Multi-datacenter requirements
•Using HashiCorp ecosystem
•Want DNS-based discovery

When Other Tools May Be Better

•Kubernetes-only (use native K8s + Istio/Linkerd)
•Only need KV store (use etcd)
•Complex coordination patterns (use ZooKeeper)
•Minimal operational overhead priority
•Single datacenter, simple needs

What's Next:

Page Complete