Tls And Encryption In Transit - Learning Module

Loading content...

0/273

mTLS for Services

Trust No Service

In traditional TLS, only the server presents a certificate. The client verifies the server's identity, but the server has no cryptographic proof of who the client is. The client might authenticate via a bearer token, API key, or username/password—but these credentials can be stolen, replayed, or misused.

Mutual TLS (mTLS) changes this dynamic: both the client and server present certificates. Each party cryptographically proves its identity to the other. No secret tokens to steal. No passwords to phish. The identity is bound to a private key that never leaves the service.

In zero-trust architectures, mTLS is the foundation of service-to-service authentication. Rather than trusting any request that arrives from the "internal network," every service verifies the cryptographic identity of every caller. A compromised service cannot impersonate another—it lacks the private key.

This page covers everything you need to implement mTLS: how it works, where it fits in your architecture, the operational challenges, and practical implementation patterns from raw configuration to service mesh automation.

What You Will Learn

By the end of this page, you will understand how mTLS works, how to implement it in Kubernetes and other environments, how service meshes automate mTLS, best practices for certificate management in mTLS scenarios, and common pitfalls to avoid.

How mTLS Works

In standard TLS, only the server authenticates to the client. In mTLS, the handshake includes an additional step where the client also presents a certificate, and the server verifies it.

Standard TLS vs mTLS:

Standard TLS (One-Way):

Client connects to server
Server presents certificate
Client validates server certificate
Encrypted channel established
Client authenticates via app-layer (token, password)

Server knows: It's talking to someone Client knows: It's talking to the right server

Mutual TLS (Two-Way):

Client connects to server
Server presents certificate
Client validates server certificate
Server requests client certificate
Client presents certificate
Server validates client certificate
Encrypted channel established

Server knows: It's talking to a verified service Client knows: It's talking to the right server

Converting Mermaid diagram...

What makes mTLS powerful:

Aspect	Token-Based Auth	mTLS
Credential type	String (JWT, API key)	Private key (never transmitted)
Replay attacks	Possible if token stolen	Impossible (key never leaves service)
Identity binding	Loose (token held by anyone)	Cryptographic (key pair per identity)
Revocation	Token blacklist, expiration	Certificate revocation, short validity
Lateral movement	Stolen token = full access	Need to steal private key from memory
Network position	Token accepted from anywhere	Can enforce source service identity

SPIFFE: Standard Identity Framework

SPIFFE (Secure Production Identity Framework for Everyone) standardizes service identity. Each service gets a SPIFFE ID (like 'spiffe://cluster.local/ns/production/sa/api-service') encoded in its certificate. SPIRE is the reference implementation. Service meshes like Istio implement SPIFFE-compatible identity.

mTLS Architecture Patterns

There are multiple ways to implement mTLS in your architecture, each with different trade-offs in complexity, flexibility, and application impact.

mTLS Implementation Patterns

•Application-Level mTLS: Each service loads its own certificate and configures TLS in application code. Maximum control, but requires changes to every service and careful certificate management.
•Sidecar Proxy (Service Mesh): Sidecar container handles all TLS. Application speaks plaintext to localhost; sidecar handles encryption. Zero application changes, but adds operational complexity.
•API Gateway/Ingress mTLS: mTLS terminates at gateway. Internal traffic may use different mechanisms. Simpler to implement but provides less granular identity.
•Network-Level (IPsec, WireGuard): Encryption at network layer, transparent to applications. Provides encryption but typically host-based identity, not workload identity.

Converting Mermaid diagram...

mTLS Pattern Comparison
Pattern	App Changes	Operations	Identity Granularity	Best For
Application mTLS	High	Medium	Per-service	Small scale, full control needed
Sidecar/Mesh	None	High	Per-workload	Kubernetes microservices
Gateway mTLS	None	Low	Per-client	External API consumers
Network-Level	None	Medium	Per-host	Legacy, VM-based workloads

Service Mesh Is the Path of Least Resistance

For Kubernetes environments with many services, service mesh (Istio, Linkerd, Consul Connect) provides mTLS with minimal effort. The mesh handles certificate issuance, rotation, and enforcement. Applications remain unaware of TLS. However, if you have few services or non-Kubernetes workloads, application-level mTLS may be simpler overall.

Implementing Application-Level mTLS

When service meshes aren't an option or finer control is needed, implementing mTLS directly in applications provides maximum flexibility.

mtls-implementation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
package main
 
import (
    "crypto/tls"
    "crypto/x509"
    "io/ioutil"
    "log"
    "net/http"
)
 
func main() {
    // Load server certificate and key
    serverCert, err := tls.LoadX509KeyPair("server.crt", "server.key")
    if err != nil {
        log.Fatal(err)
    }
 
    // Load CA certificate to verify client certificates
    caCert, err := ioutil.ReadFile("ca.crt")
    if err != nil {
        log.Fatal(err)
    }
    caCertPool := x509.NewCertPool()
    caCertPool.AppendCertsFromPEM(caCert)
 
    // Configure TLS with client certificate verification
    tlsConfig := &tls.Config{
        Certificates: []tls.Certificate{serverCert},
        ClientCAs:    caCertPool,
        // RequireAndVerifyClientCert = mTLS
        // Clients MUST present valid certificate
        ClientAuth: tls.RequireAndVerifyClientCert,
        MinVersion: tls.VersionTLS13,
    }
 
    // Create HTTPS server
    server := &http.Server{
        Addr:      ":8443",
        TLSConfig: tlsConfig,
        Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            // Access verified client certificate
            if len(r.TLS.PeerCertificates) > 0 {
                clientCert := r.TLS.PeerCertificates[0]
                log.Printf("Request from: %s", clientCert.Subject.CommonName)
                
                // Use certificate subject for authorization
                // e.g., clientCert.Subject.CommonName == "allowed-service"
            }
            w.Write([]byte("Hello, authenticated client!"))
        }),
    }
 
    log.Println("Starting mTLS server on :8443")
    // Use ListenAndServeTLS with empty strings since certs are in TLSConfig
    log.Fatal(server.ListenAndServeTLS("", ""))
}

Certificate Subject for Authorization

Authentication (proving identity) and authorization (granting access) are separate concerns. After mTLS verifies the client certificate, you still need to check if that identity is authorized for the requested action. Use the certificate's Common Name (CN), Subject Alternative Names (SAN), or SPIFFE ID to make authorization decisions.

mTLS with Service Meshes

Service meshes like Istio, Linkerd, and Consul Connect automate mTLS between all services in the mesh. This is the most operationally practical approach for Kubernetes microservices.

How mesh mTLS works:

Mesh control plane acts as certificate authority
Each workload receives automatic certificate issuance via sidecar
Certificates are short-lived (24 hours typical) and auto-rotated
All inter-service traffic is automatically encrypted with mTLS
Application code is completely unaware—it speaks plaintext to localhost

Converting Mermaid diagram...

istio-mtls-config
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
# Istio: Enable STRICT mTLS mesh-wide
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system  # Applies to entire mesh
spec:
  mtls:
    mode: STRICT  # Reject non-mTLS traffic
---
# Namespace-level mTLS policy
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: production-mtls
  namespace: production
spec:
  mtls:
    mode: STRICT
---
# Allow specific workload to accept plaintext (migration)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: legacy-service-permissive
  namespace: production
spec:
  selector:
    matchLabels:
      app: legacy-service
  mtls:
    mode: PERMISSIVE  # Accept both mTLS and plaintext
---
# Authorization policy: Only payment-service can call orders
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: orders-authz
  namespace: production
spec:
  selector:
    matchLabels:
      app: orders-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/production/sa/payment-service"
              - "cluster.local/ns/production/sa/api-gateway"
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/orders/*"]

PERMISSIVE Mode for Migration

When migrating to mTLS, start with PERMISSIVE mode which accepts both encrypted and plaintext traffic. Gradually move services to mTLS. Once all services are verified working with mTLS, switch to STRICT mode. This prevents outages during transition.

mTLS Certificate Management

mTLS requires certificates for every service—potentially thousands of them. Managing this complexity requires automation and careful planning.

mTLS Certificate Requirements

•Unique certificate per workload: Each service instance should have its own certificate. Shared certificates reduce the blast radius benefit of mTLS.
•Short validity periods: 24-hour or shorter certificates minimize exposure from compromised keys. Requires robust renewal automation.
•Private CA: Use internal CA for service certificates. Public CAs like Let's Encrypt are for public endpoints.
•SPIFFE-based identity: Encode workload identity in certificates using SPIFFE URIs (spiffe://trust-domain/path/to/workload).
•Automated issuance: Certificates should be issued automatically at workload start, not manually provisioned.
•Hot rotation: Services must reload certificates without restart. Sidecars handle this; application-level mTLS needs explicit implementation.

cert-manager-mtls
YAML (cert-manager)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
# Internal CA for mTLS certificates
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-mtls-issuer
spec:
  ca:
    secretName: internal-ca-secret
---
# Certificate for a service (referenced in pod)
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: orders-service-mtls
  namespace: production
spec:
  secretName: orders-service-mtls-secret
  
  # Short validity for mTLS
  duration: 24h
  renewBefore: 4h
  
  issuerRef:
    name: internal-mtls-issuer
    kind: ClusterIssuer
  
  # Service identity
  commonName: orders-service
  
  # SPIFFE ID in SAN extension
  uris:
    - spiffe://cluster.local/ns/production/sa/orders-service
  
  # DNS names for service discovery
  dnsNames:
    - orders-service
    - orders-service.production
    - orders-service.production.svc.cluster.local
  
  # Enable for both client and server auth
  usages:
    - digital signature
    - key encipherment
    - server auth
    - client auth
  
  privateKey:
    algorithm: ECDSA
    size: 256
---
# Mount certificate in deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-service
  namespace: production
spec:
  template:
    spec:
      containers:
        - name: orders
          image: orders:v1
          volumeMounts:
            - name: mtls-certs
              mountPath: /etc/mtls
              readOnly: true
          env:
            - name: TLS_CERT_FILE
              value: /etc/mtls/tls.crt
            - name: TLS_KEY_FILE
              value: /etc/mtls/tls.key
            - name: TLS_CA_FILE
              value: /etc/mtls/ca.crt
      volumes:
        - name: mtls-certs
          secret:
            secretName: orders-service-mtls-secret

CA Trust Distribution

Every service validating mTLS must trust the internal CA. Distribute the CA certificate via ConfigMap, mount it in all pods, or bake it into base images. Without proper CA trust, mTLS validation fails. With service meshes, this is handled automatically; for application-level mTLS, you must manage it.

Authorization with mTLS Identity

mTLS provides authentication (who is calling), but you still need authorization (are they allowed to call this endpoint). The certificate's identity becomes the basis for access control decisions.

Authorization strategies:

1. Allowlist by certificate subject:

IF request.certificate.CN IN ["payment-service", "api-gateway"]
THEN allow
ELSE deny

2. SPIFFE ID-based policies:

IF request.certificate.URI == "spiffe://cluster.local/ns/production/sa/payment"
THEN allow
ELSE deny

3. Attribute-based (certificate extensions):

IF request.certificate.extensions["department"] == "finance"
AND request.certificate.extensions["environment"] == "production"
THEN allow

4. External policy engine (OPA, Styra):

query opa.example.com/v1/data/authz {
    input: {
        caller: request.certificate.subject,
        resource: request.path,
        action: request.method
    }
}

mtls-authorization
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
package main
 
import (
    "net/http"
    "strings"
)
 
// mTLS authorization middleware
func mtlsAuthzMiddleware(allowedServices []string) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            // TLS connection info available after mTLS handshake
            if r.TLS == nil || len(r.TLS.PeerCertificates) == 0 {
                http.Error(w, "Client certificate required", http.StatusUnauthorized)
                return
            }
 
            clientCert := r.TLS.PeerCertificates[0]
            
            // Check Common Name against allowlist
            allowed := false
            for _, svc := range allowedServices {
                if clientCert.Subject.CommonName == svc {
                    allowed = true
                    break
                }
            }
 
            // Also check SPIFFE URI in SANs
            for _, uri := range clientCert.URIs {
                if strings.HasPrefix(uri.String(), "spiffe://cluster.local/ns/production/") {
                    // Parse SPIFFE ID and check authorization
                    spiffeID := uri.String()
                    if isAuthorized(spiffeID, r.URL.Path, r.Method) {
                        allowed = true
                        break
                    }
                }
            }
 
            if !allowed {
                http.Error(w, "Forbidden: service not authorized", http.StatusForbidden)
                return
            }
 
            // Add caller identity to context for logging/tracing
            r = r.WithContext(withCallerIdentity(r.Context(), clientCert.Subject.CommonName))
            
            next.ServeHTTP(w, r)
        })
    }
}
 
// Fine-grained SPIFFE-based authorization
func isAuthorized(spiffeID, path, method string) bool {
    // Example policy: only payment-service can POST to /orders
    if path == "/orders" && method == "POST" {
        return spiffeID == "spiffe://cluster.local/ns/production/sa/payment-service"
    }
    
    // Read access for any production service
    if method == "GET" {
        return strings.Contains(spiffeID, "/ns/production/")
    }
    
    return false
}

Defense in Depth

Even with mTLS authorization, maintain application-level checks. mTLS proves the caller is who they claim, but they might still be compromised. Rate limiting, input validation, and business logic authorization are still necessary. mTLS is one layer—not the only layer.

mTLS Troubleshooting

mTLS adds complexity and new failure modes. Here are common issues and how to diagnose them.

Common mTLS Issues and Solutions
Symptom	Likely Cause	Diagnosis	Solution
Connection refused	Server not requesting client cert	Check server ClientAuth setting	Set ClientAuth = RequireAndVerifyClientCert
Certificate unknown	CA not trusted	Check CA cert in trust store	Add CA cert to client RootCAs / server ClientCAs
Certificate expired	Cert validity / time skew	Check cert dates and system time	Renew cert; fix NTP synchronization
TLS handshake timeout	Firewall blocking	tcpdump / packet capture	Open required ports; check network policies
Certificate verify failed	Hostname mismatch	Check cert SANs vs connection hostname	Issue cert with correct DNS names / IPs
Bad certificate	Key/cert mismatch	Compare public key hashes	Regenerate matching key pair and certificate

mtls-debugging
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Debug mTLS connection with OpenSSL
openssl s_client -connect server:8443 \
    -cert client.crt \
    -key client.key \
    -CAfile ca.crt \
    -state -debug
 
# Check certificate validity
openssl x509 -in cert.pem -noout -dates
 
# Verify certificate chain
openssl verify -CAfile ca.crt -untrusted intermediate.crt server.crt
 
# Check if key matches certificate
openssl x509 -noout -modulus -in cert.crt | openssl md5
openssl rsa -noout -modulus -in key.pem | openssl md5
# (Output should match)
 
# View certificate details including SANs
openssl x509 -in cert.pem -noout -text | grep -A1 "Subject Alternative Name"
 
# Test mTLS with curl
curl -v \
    --cert client.crt \
    --key client.key \
    --cacert ca.crt \
    https://server:8443/api
 
# Istio: Check proxy certificate
istioctl proxy-config secret deployment/my-app -n production
 
# Istio: Verify mTLS mode
istioctl authn tls-check my-app.production production

mTLS Debugging Checklist

•Verify certificate dates: Is the certificate valid now? Check both notBefore and notAfter against current time.
•Check CA trust: Is the signing CA trusted by the verifying party? Server needs client CA; client needs server CA.
•Confirm hostname matching: Does the certificate have SANs (or CN) matching the connection hostname?
•Validate key-cert pairing: Do the private key and certificate have matching public keys?
•Check TLS version compatibility: Are both parties using compatible TLS versions and cipher suites?
•Review service mesh config: If using Istio/Linkerd, is mTLS mode correct? PeerAuthentication applied?
•Inspect network path: Are there proxies or load balancers terminating/interfering with TLS?

Service Mesh Observability

Service meshes provide rich mTLS observability. Istio's Kiali dashboard shows whether connections are mTLS-encrypted. Linkerd's viz extension displays secure vs insecure traffic. Use these tools to validate mTLS is working across your mesh before switching to STRICT mode.

Summary: mTLS for Services

We've covered mutual TLS comprehensively—from the protocol mechanics to production implementation patterns. Let's consolidate the key takeaways:

Key Takeaways

•mTLS provides cryptographic service identity — Unlike tokens that can be stolen and replayed, mTLS identity is bound to private keys that never leave the service. Compromise of one service doesn't enable impersonation of another.
•Service meshes make mTLS practical — Istio, Linkerd, and Consul handle certificate issuance, rotation, and enforcement automatically. Applications remain unaware of TLS complexity.
•Application-level mTLS provides maximum control — When meshes aren't viable, implement mTLS directly in code. Use short-lived certificates from internal CA with automated rotation.
•SPIFFE standardizes workload identity — SPIFFE IDs (spiffe://trust-domain/path) provide a standard format for expressing workload identity in certificates.
•mTLS is authentication; authorization is separate — After verifying identity, you still need policies to determine what that identity is allowed to do. Use certificate subjects or SPIFFE IDs in authorization rules.
•Migration requires PERMISSIVE mode — Move to mTLS gradually: start permissive, validate traffic is encrypted, then enforce STRICT mode.
•Troubleshooting requires visibility — OpenSSL commands, service mesh dashboards, and careful certificate inspection are essential for diagnosing mTLS issues.

Module Complete:

You've now completed the TLS and Encryption in Transit module. You understand:

TLS fundamentals and modern configuration
Where to terminate TLS in your architecture
End-to-end encryption for sensitive data
Certificate lifecycle management at scale
Mutual TLS for zero-trust service authentication

These skills form the foundation for securing all network communication in distributed systems—from external APIs to internal microservices.

Module Complete

Congratulations! You've mastered encryption in transit. You can now design TLS architectures, implement mTLS for service authentication, manage certificates at scale, and build systems where every network hop is cryptographically secured. These practices are the foundation of zero-trust networking.

mTLS for Services

Trust No Service

What You Will Learn

How mTLS Works

In standard TLS, only the server authenticates to the client. In mTLS, the handshake includes an additional step where the client also presents a certificate, and the server verifies it.

Standard TLS vs mTLS:

Standard TLS (One-Way):

Client connects to server
Server presents certificate
Client validates server certificate
Encrypted channel established
Client authenticates via app-layer (token, password)

Server knows: It's talking to someone Client knows: It's talking to the right server

Mutual TLS (Two-Way):

Client connects to server
Server presents certificate
Client validates server certificate
Server requests client certificate
Client presents certificate
Server validates client certificate
Encrypted channel established

Server knows: It's talking to a verified service Client knows: It's talking to the right server

Converting Mermaid diagram...

What makes mTLS powerful:

Aspect	Token-Based Auth	mTLS
Credential type	String (JWT, API key)	Private key (never transmitted)
Replay attacks	Possible if token stolen	Impossible (key never leaves service)
Identity binding	Loose (token held by anyone)	Cryptographic (key pair per identity)
Revocation	Token blacklist, expiration	Certificate revocation, short validity
Lateral movement	Stolen token = full access	Need to steal private key from memory
Network position	Token accepted from anywhere	Can enforce source service identity

SPIFFE: Standard Identity Framework

mTLS Architecture Patterns

There are multiple ways to implement mTLS in your architecture, each with different trade-offs in complexity, flexibility, and application impact.

mTLS Implementation Patterns

•Application-Level mTLS: Each service loads its own certificate and configures TLS in application code. Maximum control, but requires changes to every service and careful certificate management.
•Sidecar Proxy (Service Mesh): Sidecar container handles all TLS. Application speaks plaintext to localhost; sidecar handles encryption. Zero application changes, but adds operational complexity.
•API Gateway/Ingress mTLS: mTLS terminates at gateway. Internal traffic may use different mechanisms. Simpler to implement but provides less granular identity.
•Network-Level (IPsec, WireGuard): Encryption at network layer, transparent to applications. Provides encryption but typically host-based identity, not workload identity.

Converting Mermaid diagram...

mTLS Pattern Comparison
Pattern	App Changes	Operations	Identity Granularity	Best For
Application mTLS	High	Medium	Per-service	Small scale, full control needed
Sidecar/Mesh	None	High	Per-workload	Kubernetes microservices
Gateway mTLS	None	Low	Per-client	External API consumers
Network-Level	None	Medium	Per-host	Legacy, VM-based workloads

Service Mesh Is the Path of Least Resistance

Implementing Application-Level mTLS

When service meshes aren't an option or finer control is needed, implementing mTLS directly in applications provides maximum flexibility.

mtls-implementation
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
package main
 
import (
    "crypto/tls"
    "crypto/x509"
    "io/ioutil"
    "log"
    "net/http"
)
 
func main() {
    // Load server certificate and key
    serverCert, err := tls.LoadX509KeyPair("server.crt", "server.key")
    if err != nil {
        log.Fatal(err)
    }
 
    // Load CA certificate to verify client certificates
    caCert, err := ioutil.ReadFile("ca.crt")
    if err != nil {
        log.Fatal(err)
    }
    caCertPool := x509.NewCertPool()
    caCertPool.AppendCertsFromPEM(caCert)
 
    // Configure TLS with client certificate verification
    tlsConfig := &tls.Config{
        Certificates: []tls.Certificate{serverCert},
        ClientCAs:    caCertPool,
        // RequireAndVerifyClientCert = mTLS
        // Clients MUST present valid certificate
        ClientAuth: tls.RequireAndVerifyClientCert,
        MinVersion: tls.VersionTLS13,
    }
 
    // Create HTTPS server
    server := &http.Server{
        Addr:      ":8443",
        TLSConfig: tlsConfig,
        Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            // Access verified client certificate
            if len(r.TLS.PeerCertificates) > 0 {
                clientCert := r.TLS.PeerCertificates[0]
                log.Printf("Request from: %s", clientCert.Subject.CommonName)
                
                // Use certificate subject for authorization
                // e.g., clientCert.Subject.CommonName == "allowed-service"
            }
            w.Write([]byte("Hello, authenticated client!"))
        }),
    }
 
    log.Println("Starting mTLS server on :8443")
    // Use ListenAndServeTLS with empty strings since certs are in TLSConfig
    log.Fatal(server.ListenAndServeTLS("", ""))
}

Certificate Subject for Authorization

mTLS with Service Meshes

Service meshes like Istio, Linkerd, and Consul Connect automate mTLS between all services in the mesh. This is the most operationally practical approach for Kubernetes microservices.

How mesh mTLS works:

Mesh control plane acts as certificate authority
Each workload receives automatic certificate issuance via sidecar
Certificates are short-lived (24 hours typical) and auto-rotated
All inter-service traffic is automatically encrypted with mTLS
Application code is completely unaware—it speaks plaintext to localhost

Converting Mermaid diagram...

istio-mtls-config
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
# Istio: Enable STRICT mTLS mesh-wide
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system  # Applies to entire mesh
spec:
  mtls:
    mode: STRICT  # Reject non-mTLS traffic
---
# Namespace-level mTLS policy
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: production-mtls
  namespace: production
spec:
  mtls:
    mode: STRICT
---
# Allow specific workload to accept plaintext (migration)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: legacy-service-permissive
  namespace: production
spec:
  selector:
    matchLabels:
      app: legacy-service
  mtls:
    mode: PERMISSIVE  # Accept both mTLS and plaintext
---
# Authorization policy: Only payment-service can call orders
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: orders-authz
  namespace: production
spec:
  selector:
    matchLabels:
      app: orders-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/production/sa/payment-service"
              - "cluster.local/ns/production/sa/api-gateway"
      to:
        - operation:
            methods: ["GET", "POST"]
            paths: ["/orders/*"]

PERMISSIVE Mode for Migration

mTLS Certificate Management

mTLS requires certificates for every service—potentially thousands of them. Managing this complexity requires automation and careful planning.

mTLS Certificate Requirements

•Unique certificate per workload: Each service instance should have its own certificate. Shared certificates reduce the blast radius benefit of mTLS.
•Short validity periods: 24-hour or shorter certificates minimize exposure from compromised keys. Requires robust renewal automation.
•Private CA: Use internal CA for service certificates. Public CAs like Let's Encrypt are for public endpoints.
•SPIFFE-based identity: Encode workload identity in certificates using SPIFFE URIs (spiffe://trust-domain/path/to/workload).
•Automated issuance: Certificates should be issued automatically at workload start, not manually provisioned.
•Hot rotation: Services must reload certificates without restart. Sidecars handle this; application-level mTLS needs explicit implementation.

cert-manager-mtls
YAML (cert-manager)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
# Internal CA for mTLS certificates
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-mtls-issuer
spec:
  ca:
    secretName: internal-ca-secret
---
# Certificate for a service (referenced in pod)
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: orders-service-mtls
  namespace: production
spec:
  secretName: orders-service-mtls-secret
  
  # Short validity for mTLS
  duration: 24h
  renewBefore: 4h
  
  issuerRef:
    name: internal-mtls-issuer
    kind: ClusterIssuer
  
  # Service identity
  commonName: orders-service
  
  # SPIFFE ID in SAN extension
  uris:
    - spiffe://cluster.local/ns/production/sa/orders-service
  
  # DNS names for service discovery
  dnsNames:
    - orders-service
    - orders-service.production
    - orders-service.production.svc.cluster.local
  
  # Enable for both client and server auth
  usages:
    - digital signature
    - key encipherment
    - server auth
    - client auth
  
  privateKey:
    algorithm: ECDSA
    size: 256
---
# Mount certificate in deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-service
  namespace: production
spec:
  template:
    spec:
      containers:
        - name: orders
          image: orders:v1
          volumeMounts:
            - name: mtls-certs
              mountPath: /etc/mtls
              readOnly: true
          env:
            - name: TLS_CERT_FILE
              value: /etc/mtls/tls.crt
            - name: TLS_KEY_FILE
              value: /etc/mtls/tls.key
            - name: TLS_CA_FILE
              value: /etc/mtls/ca.crt
      volumes:
        - name: mtls-certs
          secret:
            secretName: orders-service-mtls-secret

CA Trust Distribution

Authorization with mTLS Identity

mTLS provides authentication (who is calling), but you still need authorization (are they allowed to call this endpoint). The certificate's identity becomes the basis for access control decisions.

Authorization strategies:

1. Allowlist by certificate subject:

IF request.certificate.CN IN ["payment-service", "api-gateway"]
THEN allow
ELSE deny

2. SPIFFE ID-based policies:

IF request.certificate.URI == "spiffe://cluster.local/ns/production/sa/payment"
THEN allow
ELSE deny

3. Attribute-based (certificate extensions):

IF request.certificate.extensions["department"] == "finance"
AND request.certificate.extensions["environment"] == "production"
THEN allow

4. External policy engine (OPA, Styra):

query opa.example.com/v1/data/authz {
    input: {
        caller: request.certificate.subject,
        resource: request.path,
        action: request.method
    }
}

mtls-authorization
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
package main
 
import (
    "net/http"
    "strings"
)
 
// mTLS authorization middleware
func mtlsAuthzMiddleware(allowedServices []string) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            // TLS connection info available after mTLS handshake
            if r.TLS == nil || len(r.TLS.PeerCertificates) == 0 {
                http.Error(w, "Client certificate required", http.StatusUnauthorized)
                return
            }
 
            clientCert := r.TLS.PeerCertificates[0]
            
            // Check Common Name against allowlist
            allowed := false
            for _, svc := range allowedServices {
                if clientCert.Subject.CommonName == svc {
                    allowed = true
                    break
                }
            }
 
            // Also check SPIFFE URI in SANs
            for _, uri := range clientCert.URIs {
                if strings.HasPrefix(uri.String(), "spiffe://cluster.local/ns/production/") {
                    // Parse SPIFFE ID and check authorization
                    spiffeID := uri.String()
                    if isAuthorized(spiffeID, r.URL.Path, r.Method) {
                        allowed = true
                        break
                    }
                }
            }
 
            if !allowed {
                http.Error(w, "Forbidden: service not authorized", http.StatusForbidden)
                return
            }
 
            // Add caller identity to context for logging/tracing
            r = r.WithContext(withCallerIdentity(r.Context(), clientCert.Subject.CommonName))
            
            next.ServeHTTP(w, r)
        })
    }
}
 
// Fine-grained SPIFFE-based authorization
func isAuthorized(spiffeID, path, method string) bool {
    // Example policy: only payment-service can POST to /orders
    if path == "/orders" && method == "POST" {
        return spiffeID == "spiffe://cluster.local/ns/production/sa/payment-service"
    }
    
    // Read access for any production service
    if method == "GET" {
        return strings.Contains(spiffeID, "/ns/production/")
    }
    
    return false
}

Defense in Depth

mTLS Troubleshooting

mTLS adds complexity and new failure modes. Here are common issues and how to diagnose them.

Common mTLS Issues and Solutions
Symptom	Likely Cause	Diagnosis	Solution
Connection refused	Server not requesting client cert	Check server ClientAuth setting	Set ClientAuth = RequireAndVerifyClientCert
Certificate unknown	CA not trusted	Check CA cert in trust store	Add CA cert to client RootCAs / server ClientCAs
Certificate expired	Cert validity / time skew	Check cert dates and system time	Renew cert; fix NTP synchronization
TLS handshake timeout	Firewall blocking	tcpdump / packet capture	Open required ports; check network policies
Certificate verify failed	Hostname mismatch	Check cert SANs vs connection hostname	Issue cert with correct DNS names / IPs
Bad certificate	Key/cert mismatch	Compare public key hashes	Regenerate matching key pair and certificate

mtls-debugging
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# Debug mTLS connection with OpenSSL
openssl s_client -connect server:8443 \
    -cert client.crt \
    -key client.key \
    -CAfile ca.crt \
    -state -debug
 
# Check certificate validity
openssl x509 -in cert.pem -noout -dates
 
# Verify certificate chain
openssl verify -CAfile ca.crt -untrusted intermediate.crt server.crt
 
# Check if key matches certificate
openssl x509 -noout -modulus -in cert.crt | openssl md5
openssl rsa -noout -modulus -in key.pem | openssl md5
# (Output should match)
 
# View certificate details including SANs
openssl x509 -in cert.pem -noout -text | grep -A1 "Subject Alternative Name"
 
# Test mTLS with curl
curl -v \
    --cert client.crt \
    --key client.key \
    --cacert ca.crt \
    https://server:8443/api
 
# Istio: Check proxy certificate
istioctl proxy-config secret deployment/my-app -n production
 
# Istio: Verify mTLS mode
istioctl authn tls-check my-app.production production

mTLS Debugging Checklist

•Verify certificate dates: Is the certificate valid now? Check both notBefore and notAfter against current time.
•Check CA trust: Is the signing CA trusted by the verifying party? Server needs client CA; client needs server CA.
•Confirm hostname matching: Does the certificate have SANs (or CN) matching the connection hostname?
•Validate key-cert pairing: Do the private key and certificate have matching public keys?
•Check TLS version compatibility: Are both parties using compatible TLS versions and cipher suites?
•Review service mesh config: If using Istio/Linkerd, is mTLS mode correct? PeerAuthentication applied?
•Inspect network path: Are there proxies or load balancers terminating/interfering with TLS?

Service Mesh Observability

Summary: mTLS for Services

We've covered mutual TLS comprehensively—from the protocol mechanics to production implementation patterns. Let's consolidate the key takeaways:

Key Takeaways

•mTLS provides cryptographic service identity — Unlike tokens that can be stolen and replayed, mTLS identity is bound to private keys that never leave the service. Compromise of one service doesn't enable impersonation of another.
•Service meshes make mTLS practical — Istio, Linkerd, and Consul handle certificate issuance, rotation, and enforcement automatically. Applications remain unaware of TLS complexity.
•Application-level mTLS provides maximum control — When meshes aren't viable, implement mTLS directly in code. Use short-lived certificates from internal CA with automated rotation.
•SPIFFE standardizes workload identity — SPIFFE IDs (spiffe://trust-domain/path) provide a standard format for expressing workload identity in certificates.
•mTLS is authentication; authorization is separate — After verifying identity, you still need policies to determine what that identity is allowed to do. Use certificate subjects or SPIFFE IDs in authorization rules.
•Migration requires PERMISSIVE mode — Move to mTLS gradually: start permissive, validate traffic is encrypted, then enforce STRICT mode.
•Troubleshooting requires visibility — OpenSSL commands, service mesh dashboards, and careful certificate inspection are essential for diagnosing mTLS issues.

Module Complete:

You've now completed the TLS and Encryption in Transit module. You understand:

TLS fundamentals and modern configuration
Where to terminate TLS in your architecture
End-to-end encryption for sensitive data
Certificate lifecycle management at scale
Mutual TLS for zero-trust service authentication

These skills form the foundation for securing all network communication in distributed systems—from external APIs to internal microservices.

Module Complete