API Gateway Solutions - Learning Module

Loading content...

0/273

Envoy: Service Proxy

The Proxy That Powers the Cloud-Native Ecosystem

While we've discussed API Gateway products built on Envoy, understanding Envoy itself is essential for any serious distributed systems practitioner. Envoy isn't just another proxy—it's become the foundational data plane for modern cloud-native infrastructure.

Born at Lyft in 2016 to solve service mesh challenges, Envoy is now a CNCF graduated project powering an extraordinary range of systems: Istio, Ambassador, Contour, AWS App Mesh, Gloo Edge, Consul Connect, and many more. Whether you choose Envoy directly or through an abstraction, understanding its architecture is invaluable.

This page provides a comprehensive exploration of Envoy's architecture, configuration model, extensibility mechanisms, and direct deployment scenarios.

What You Will Learn

By the end of this page, you will understand Envoy's architecture and design philosophy, the xDS dynamic configuration APIs, filter chains and extensibility, performance characteristics, and when to use Envoy directly versus through abstraction layers.

Envoy Architecture Overview

Design Goals

Envoy was designed with specific goals that differentiate it from traditional proxies:

The network should be transparent to applications — Applications don't configure proxies; the mesh handles it
Configuration is fully dynamic — No restarts, no config reloads with connection drops
Observability is a first-class feature — Metrics, tracing, logging built-in
Extensibility without recompilation — Filters, WASM, Lua

Core Concepts

Envoy Core Terminology
Concept	Description	Example
Downstream	Client connecting to Envoy	Browser, mobile app, another service
Upstream	Service Envoy proxies to	Your backend API server
Listener	Named network location Envoy binds to	0.0.0.0:8080
Cluster	Group of logically similar upstreams	'user-service' cluster
Endpoint	IP/port of a specific upstream instance	10.0.1.5:8080
Filter	Processing unit in request/response path	Router, AuthN, Rate Limit
Route	Rules mapping requests to clusters	/api/users → user-service

┌─────────────────────────────────────────────────────────────────────────────────┐
│                              ENVOY PROXY                                        │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                 │
│   Downstream (Client)                                                           │
│         │                                                                       │
│         ▼                                                                       │
│   ┌─────────────────┐                                                          │
│   │    LISTENER     │  ← Bind address (0.0.0.0:8080)                           │
│   │  (Filter Chain) │                                                          │
│   └────────┬────────┘                                                          │
│            │                                                                    │
│            ▼                                                                    │
│   ┌───────────────────────────────────────────────────────────┐                │
│   │                    FILTER CHAIN                           │                │
│   │  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐ │                │
│   │  │  TLS     │→ │  HTTP    │→ │  ExtAuthz │→ │  Router  │ │                │
│   │  │  Filter  │  │  Conn Mgr│  │  Filter  │  │  Filter  │ │                │
│   │  └──────────┘  └──────────┘  └──────────┘  └──────────┘ │                │
│   └───────────────────────────────────────────────────────────┘                │
│            │                                                                    │
│            ▼                                                                    │
│   ┌─────────────────┐     (Route Match: /api/users → user-service)             │
│   │     ROUTES      │                                                          │
│   └────────┬────────┘                                                          │
│            │                                                                    │
│            ▼                                                                    │
│   ┌─────────────────┐                                                          │
│   │    CLUSTER      │  ← user-service                                          │
│   │  (Load Balance) │  ← Round-robin, least-request, etc.                      │
│   └────────┬────────┘                                                          │
│            │                                                                    │
│            ▼                                                                    │
│   ┌─────────────────┐                                                          │
│   │   ENDPOINTS     │  ← 10.0.1.5:8080, 10.0.1.6:8080                          │
│   │  (Health Check) │  ← Active/passive health checking                        │
│   └────────┬────────┘                                                          │
│            │                                                                    │
│            ▼                                                                    │
│   Upstream (Backend Service)                                                    │
│                                                                                 │
└─────────────────────────────────────────────────────────────────────────────────┘

Mental Model

Think of Envoy as: Listeners receive connections → Filter chains process requests → Routes select clusters → Clusters load-balance across endpoints. Each step is configurable and observable.

The xDS Discovery APIs

The xDS APIs are Envoy's killer feature. They enable fully dynamic configuration without restarts—something revolutionary when Envoy launched.

The Discovery Services

xDS API Types
API	Name	Configures	Update Impact
LDS	Listener Discovery	Listeners, filter chains	Adds/removes ports
RDS	Route Discovery	Virtual hosts, routes	Routing rule changes
CDS	Cluster Discovery	Upstream clusters	New services
EDS	Endpoint Discovery	Cluster endpoints	Instance scaling
SDS	Secret Discovery	TLS certificates	Cert rotation
VHDS	Virtual Host Discovery	Per-route virtual hosts	Large route tables
RTDS	Runtime Discovery	Runtime configuration	Feature flags

xDS Server Architecture

Envoy connects to an xDS server (control plane) via gRPC streaming:

┌─────────────────┐        gRPC Stream        ┌─────────────────┐
│                 │  ←──── Discovery ────────  │                 │
│  Envoy Proxy    │        Requests            │   xDS Server    │
│  (Data Plane)   │  ─────── xDS ────────────► │  (Control Plane)│
│                 │        Responses           │                 │
└─────────────────┘                            └─────────────────┘

xDS servers include:

Istio Pilot — Kubernetes-aware, service mesh focused
Ambassador — Kubernetes CRD to xDS translator
Consul Connect — HashiCorp service discovery integration
go-control-plane — Reference implementation for custom control planes

Aggregated xDS (ADS)

ADS multiplexes all xDS types over a single gRPC stream, ensuring configuration consistency and ordering:

# Envoy bootstrap config with ADS
node:
  id: envoy-1
  cluster: edge-cluster

dynamic_resources:
  ads_config:
    api_type: GRPC
    transport_api_version: V3
    grpc_services:
      - envoy_grpc:
          cluster_name: xds-cluster
  cds_config:
    ads: {}
  lds_config:
    ads: {}

Delta xDS

xDS v3 supports delta (incremental) updates—only changed resources are transmitted. This is critical for large deployments with thousands of routes or endpoints where full config transmission would be prohibitively expensive.

Filter Chains and Extensibility

Envoy's filter chain model provides extensibility at multiple levels.

Filter Types

Envoy Filter Categories
Type	Level	Examples	Use Case
Listener Filters	L4 (connection)	TLS Inspector, Proxy Protocol	Multi-protocol detection
Network Filters	L4 (stream)	TCP Proxy, HTTP Connection Manager	Protocol handling
HTTP Filters	L7 (request)	Router, ExtAuthz, CORS, Rate Limit	Request processing

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
listeners:
  - name: http_listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 8080
    filter_chains:
      - filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_http
              codec_type: AUTO
              route_config:
                name: local_route
                virtual_hosts:
                  - name: backend
                    domains: ["*"]
                    routes:
                      - match:
                          prefix: "/"
                        route:
                          cluster: backend_service
              http_filters:
                # 1. External Authentication
                - name: envoy.filters.http.ext_authz
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
                    grpc_service:
                      envoy_grpc:
                        cluster_name: auth_service
                    transport_api_version: V3
                
                # 2. Rate Limiting
                - name: envoy.filters.http.ratelimit
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit
                    domain: production
                    rate_limit_service:
                      grpc_service:
                        envoy_grpc:
                          cluster_name: rate_limit_service
                
                # 3. Router (terminal filter - must be last)
                - name: envoy.filters.http.router
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

Extension Mechanisms

Lua Filters — Inline scripting for simple transformations:

- name: envoy.filters.http.lua
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
    inline_code: |
      function envoy_on_request(handle)
        handle:headers():add("X-Custom-Header", "value")
      end

WASM (WebAssembly) Filters — Portable, sandboxed extensions:

Write in C++, Rust, Go, AssemblyScript
Sandboxed execution
Portable across Envoy deployments
Updated without Envoy restart

Lua vs. WASM vs. External

Lua: Simple header manipulation, request logging (low overhead). WASM: Complex logic, portable across platforms (medium overhead). External services (ExtAuthz, RateLimit): Full-featured, language-agnostic (network overhead). Choose based on complexity and latency requirements.

Built-in Observability

Observability isn't an add-on in Envoy—it's a core design principle.

Metrics

Envoy exposes comprehensive metrics via /stats endpoint in Prometheus format:

Per-cluster metrics: upstream_rq_total, upstream_cx_total, upstream_rq_time
Per-route metrics: Rate, latency percentiles
Per-listener metrics: downstream_cx_total, downstream_rq_time
Health check metrics: Per-endpoint health status

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
stats_sinks:
  - name: envoy.stat_sinks.statsd
    typed_config:
      "@type": type.googleapis.com/envoy.config.metrics.v3.StatsdSink
      address:
        socket_address:
          address: statsd
          port_value: 8125
          
  - name: envoy.stat_sinks.metrics_service
    typed_config:
      "@type": type.googleapis.com/envoy.config.metrics.v3.MetricsServiceConfig
      grpc_service:
        envoy_grpc:
          cluster_name: metrics_cluster
 
stats_config:
  stats_tags:
    - tag_name: service
      regex: "^cluster\.(.*?)\."
  use_all_default_tags: true

Distributed Tracing

Envoy integrates with major tracing systems:

Generates trace headers — x-request-id, b3 propagation, W3C Trace Context
Reports spans — Zipkin, Jaeger, Datadog, OpenTelemetry
Configurable sampling — Per-route or global sampling rates

Access Logging

Highly configurable access logs with format customization:

access_log:
  - name: envoy.access_loggers.file
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
      path: /dev/stdout
      log_format:
        json_format:
          time: "%START_TIME%"
          method: "%REQ(:METHOD)%"
          path: "%REQ(:PATH)%"
          status: "%RESPONSE_CODE%"
          duration: "%DURATION%"
          upstream: "%UPSTREAM_HOST%"
          trace_id: "%REQ(X-REQUEST-ID)%"

Performance Characteristics

Envoy's C++ implementation and event-driven architecture deliver exceptional performance.

Envoy Performance Benchmarks (Reference)
Metric	Value	Conditions
Added Latency	< 1ms P50, < 2ms P99	Simple routing, no filters
Throughput	~100K RPS per core	Small payloads, HTTP/1.1
Memory per connection	~20KB	HTTP/1.1, minimal config
Connection capacity	50K+ per instance	Depends on memory
Config reload	~0ms (xDS)	No connection drops

Memory Considerations

Envoy memory scales with:

Connection count — ~20KB per HTTP/1.1, ~40KB per HTTP/2
Route table size — Large route configs consume significant memory
Stats cardinality — High-cardinality stats (per-cluster, per-route) add up
Buffer sizes — Request/response buffering for transformations

CPU Optimization

Disable unnecessary filters — Each filter adds processing overhead
Use HTTP/2 — Multiplexing reduces connection overhead
Tune worker threads — Default matches CPU count; tune for workload
Minimize logging — Access logging at volume impacts CPU

Stats Cardinality Explosion

Envoy generates stats per-cluster, per-route, per-endpoint. With thousands of routes and endpoints, stats memory can explode. Use stats_matcher to limit stats collection in high-cardinality environments.

Direct Envoy Usage Scenarios

While many use Envoy through abstraction layers, there are compelling scenarios for direct usage:

When to Use Envoy Directly

Direct Envoy Use Cases

•Custom control plane — Building your own xDS server for specific requirements
•Maximum performance — Eliminate abstraction layer overhead
•Non-Kubernetes — VMs, bare metal, custom orchestration
•Edge proxy — Simple edge routing without Kubernetes
•Learning/experimentation — Understanding Envoy's native config

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901
 
static_resources:
  listeners:
    - name: http_listener
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 8080
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                stat_prefix: ingress
                route_config:
                  name: local_routes
                  virtual_hosts:
                    - name: services
                      domains: ["*"]
                      routes:
                        - match: { prefix: "/api/users" }
                          route: { cluster: user_service }
                        - match: { prefix: "/api/orders" }
                          route: { cluster: order_service }
                http_filters:
                  - name: envoy.filters.http.router
                    typed_config:
                      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
 
  clusters:
    - name: user_service
      connect_timeout: 5s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: user_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: user-service
                      port_value: 8080
      health_checks:
        - timeout: 5s
          interval: 10s
          unhealthy_threshold: 3
          healthy_threshold: 2
          http_health_check:
            path: /health
 
    - name: order_service
      connect_timeout: 5s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      load_assignment:
        cluster_name: order_service
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: order-service
                      port_value: 8080

Summary: Envoy Service Proxy

Key Takeaways

•Foundation of Cloud-Native — Envoy powers Istio, Ambassador, Contour, and many more.
•xDS is Revolutionary — Fully dynamic config via streaming APIs; no restarts ever.
•Filter Chain Architecture — Composable processing pipeline for L4 and L7.
•Observability Built-In — Metrics, tracing, logging as first-class features.
•Performance Excellence — Sub-millisecond latency, high throughput, efficient memory use.
•Direct or Abstracted — Use directly for control, or via Ambassador/Istio for convenience.

What's Next:

Having explored the major API Gateway technologies—Kong, AWS API Gateway, Ambassador, and Envoy—we'll conclude with a decision framework: How to Choose an API Gateway for your specific requirements.

Page Complete

You now understand Envoy's architecture, xDS configuration model, filter chains, observability features, and performance characteristics. This knowledge applies whether using Envoy directly or through abstraction layers.