Secure Shell (SSH)

SSH keys are the foundation of secure authentication in modern infrastructure. Unlike passwords—which can be guessed, stolen through phishing, or compromised through server breaches—cryptographic keys provide authentication that scales from a single developer's laptop to global infrastructure with millions of servers.

But this power comes with responsibility. A compromised SSH key is a complete identity theft. Poorly managed keys accumulate across systems, creating audit nightmares and security vulnerabilities. Organizations have discovered thousands of untracked SSH keys granting access to critical systems, each one a potential entry point for attackers.

SSH supports multiple key types, each with different security properties, performance characteristics, and compatibility considerations.

The Key Pair Concept:

SSH keys come in pairs:

• Private key: Kept secret, proves identity through signatures
• Public key: Shared freely, used to verify signatures

Anyone with the public key can verify that a signature was created by the corresponding private key—but cannot create signatures themselves.

Modern Key Types:

Generating Keys:

# Ed25519 (recommended)
ssh-keygen -t ed25519 -C "your_email@example.com"
# Creates: ~/.ssh/id_ed25519, ~/.ssh/id_ed25519.pub

# Ed25519 with specific filename
ssh-keygen -t ed25519 -f ~/.ssh/work_key -C "work laptop 2024"

# RSA (when ed25519 not supported)
ssh-keygen -t rsa -b 4096 -C "your_email@example.com"
# Creates: ~/.ssh/id_rsa, ~/.ssh/id_rsa.pub

# ECDSA
ssh-keygen -t ecdsa -b 384 -C "your_email@example.com"
# Creates: ~/.ssh/id_ecdsa, ~/.ssh/id_ecdsa.pub

# FIDO2/Hardware key (requires security key)
ssh-keygen -t ed25519-sk -C "hardware key 2024"
# Creates: ~/.ssh/id_ed25519_sk, ~/.ssh/id_ed25519_sk.pub

Generation Options:

SSH keys can be stored in several formats, with modern OpenSSH using an encrypted format that resists brute-force attacks on passphrases.

Private Key Formats:

OpenSSH Format Advantages:

The new OpenSSH format uses bcrypt as the key derivation function:

• Resistant to brute-force passphrase attacks
• Configurable difficulty (-a rounds)
• Supports Ed25519 (which PEM format doesn't)

Converting Formats:

# Convert old PEM to new OpenSSH format
ssh-keygen -p -o -f ~/.ssh/id_rsa
# -p: change passphrase (or convert)
# -o: use new format

# Convert to PEM format (for compatibility)
ssh-keygen -p -m PEM -f ~/.ssh/id_rsa

# Convert PuTTY key to OpenSSH (requires puttygen)
puttygen keyfile.ppk -O private-openssh -o id_rsa

# Extract public key from private key
ssh-keygen -y -f ~/.ssh/id_ed25519 > ~/.ssh/id_ed25519.pub

File Permissions:

SSH enforces strict permissions on key files:

The authorized_keys file on servers lists public keys allowed to authenticate as that user. Understanding its format and options is essential for secure, flexible access control.

File Location:

# Default location:
~/.ssh/authorized_keys

# Can be configured in sshd_config:
AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2

# Centralized location:
AuthorizedKeysFile /etc/ssh/authorized_keys/%u
# %u expands to username

Basic Format:

[options] key-type base64-key comment

Example File:

# Simple entries (no options)
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGyz... alice@laptop
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCz... bob@desktop

# With options
command="/usr/local/bin/backup.sh" ssh-ed25519 AAAA... backup@server
from="192.168.1.*,10.0.0.0/8" ssh-ed25519 AAAA... admin@internal

Available Options:

Practical Examples:

# Backup key: only runs backup, no shell, no forwarding
command="/usr/local/bin/backup.sh",no-port-forwarding,no-X11-forwarding,no-pty ssh-ed25519 AAAA... backup-agent

# Restricted key with specific port forward allowed
restrict,permitopen="database:5432" ssh-ed25519 AAAA... db-access-user

# Time-limited access key
expiry-time="20240630",from="10.0.0.0/8" ssh-ed25519 AAAA... contractor

# Git server: only git commands
command="/usr/bin/git-shell -c \"$SSH_ORIGINAL_COMMAND\"",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-ed25519 AAAA... git-user

# Multiple restrictions combined
from="192.168.1.0/24",command="df -h",no-pty ssh-ed25519 AAAA... monitoring-agent

Getting public keys to servers—and keeping them current—is one of the most challenging aspects of SSH key management at scale.

Manual Distribution:

# Copy public key to server (ssh-copy-id)
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@server
# Appends to ~/.ssh/authorized_keys on server

# Manual method (when ssh-copy-id unavailable)
cat ~/.ssh/id_ed25519.pub | ssh user@server \
  'mkdir -p ~/.ssh && chmod 700 ~/.ssh && cat >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys'

# Copy to multiple servers
for server in server1 server2 server3; do
  ssh-copy-id -i ~/.ssh/id_ed25519.pub user@$server
done

Configuration Management:

Configuration management tools automate key distribution:

LDAP/Directory Integration:

PublicKeys can be stored in LDAP directories:

# sshd_config:
AuthorizedKeysCommand /usr/bin/sss_ssh_authorizedkeys
AuthorizedKeysCommandUser nobody

# Fetches keys from LDAP/AD via SSSD
# User's public keys stored as sshPublicKey attribute

GitHub/GitLab Keys:

Public services expose user keys:

# Fetch keys from GitHub
curl https://github.com/username.keys

# Use in authorized_keys via AuthorizedKeysCommand
#!/bin/bash
curl -s "https://github.com/$1.keys"

# sshd_config:
AuthorizedKeysCommand /usr/local/bin/fetch-github-keys %u
AuthorizedKeysCommandUser nobody

SSH certificates fundamentally solve the key distribution problem. Instead of distributing individual public keys, you distribute a CA public key once, then issue signed certificates to users.

Certificate-Based Infrastructure:

Creating a User CA:

# Generate CA key pair (protect the private key!)
ssh-keygen -t ed25519 -f /secure/path/user_ca -C "SSH User CA"

# No passphrase for automated issuance (secure the private key physically)
# Or with passphrase for manual issuance

Issuing User Certificates:

# Sign a user's public key
ssh-keygen -s /secure/path/user_ca \
    -I "alice@example.com" \
    -n alice,ops-team \
    -V +1d \
    ~/.ssh/id_ed25519.pub

# Options:
#   -s: CA private key for signing
#   -I: Certificate identity (for audit logs)
#   -n: Principals (usernames user can authenticate as)
#   -V: Validity period (+1d = one day from now)
#
# Output: ~/.ssh/id_ed25519-cert.pub

Server Configuration:

# /etc/ssh/sshd_config

# Trust certificates signed by this CA
TrustedUserCAKeys /etc/ssh/user_ca.pub

# Optional: restrict to specific principals
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u
# File contents: one principal per line
# /etc/ssh/auth_principals/deploy:
#   ops-team
#   ci-pipeline

User Configuration:

# Certificate is automatically used if present alongside the key
# ~/.ssh/id_ed25519
# ~/.ssh/id_ed25519-cert.pub
# SSH client automatically presents both

# Or explicitly specify:
ssh -i ~/.ssh/id_ed25519 user@server
# SSH sends id_ed25519-cert.pub if it exists

Host key certificates solve the "Trust on First Use" problem. Instead of manually verifying host key fingerprints, clients trust certificates signed by a known CA.

Creating a Host CA:

# Generate host CA (separate from user CA)
ssh-keygen -t ed25519 -f /secure/path/host_ca -C "SSH Host CA"

Signing Host Keys:

# On each server, sign the host key
ssh-keygen -s /secure/path/host_ca \
    -I "server.example.com" \
    -h \
    -n server.example.com,server,10.0.1.50 \
    -V +52w \
    /etc/ssh/ssh_host_ed25519_key.pub

# Options:
#   -h: This is a host certificate (not user)
#   -n: Hostnames this cert is valid for
#
# Output: /etc/ssh/ssh_host_ed25519_key-cert.pub

Server Configuration:

# /etc/ssh/sshd_config

# Present the host certificate
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

# Can have multiple host keys/certificates
HostKey /etc/ssh/ssh_host_ed25519_key
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

Client Configuration:

# ~/.ssh/known_hosts
# Instead of individual host keys, trust the CA:
@cert-authority *.example.com ssh-ed25519 AAAA...host-ca-public-key...

# All hosts matching *.example.com with valid CA-signed certs are trusted
# No more "authenticity can't be established" prompts!

# Or in /etc/ssh/ssh_known_hosts for system-wide trust

Keys don't last forever. Employees leave, keys get compromised, security policies require periodic rotation. Having robust rotation and revocation processes is essential.

Key Rotation Strategies:

SSH Revoked Keys (Key Revocation List):

# /etc/ssh/sshd_config:
RevokedKeys /etc/ssh/revoked_keys

# The revoked_keys file is a Key Revocation List (KRL)
# Create/update using ssh-keygen:

# Create KRL from public key(s)
ssh-keygen -k -f /etc/ssh/revoked_keys \
    compromised_key.pub former_employee.pub

# Add to existing KRL
ssh-keygen -k -u -f /etc/ssh/revoked_keys \
    another_key.pub

# Revoke by serial number (for certificates)
ssh-keygen -k -f /etc/ssh/revoked_keys -z 1234

# Revoke by key ID (for certificates)
ssh-keygen -k -f /etc/ssh/revoked_keys \
    -s user_ca -I "user@example.com"

Certificate Revocation:

# Revoke a certificate by serial number
ssh-keygen -k -f /etc/ssh/revoked_keys -s user_ca -z 42

# Revoke all certificates with a specific key ID
ssh-keygen -k -f /etc/ssh/revoked_keys -s user_ca \
    -I "alice@example.com"

# The KRL must be distributed to all servers
# This is why short-lived certificates are often preferred.

Rotation Best Practices:

Large organizations face scale challenges: thousands of users, hundreds of thousands of servers, compliance requirements, and audit demands. Dedicated SSH key management solutions address these needs.

Key Management Challenges at Scale:

Enterprise Solutions:

1. SSH Certificate Authorities (DIY) Build your own CA infrastructure with automation:

• Vault (HashiCorp) SSH secrets engine
• Custom CA with certificate issuance API
• Integration with SSO for authentication

# HashiCorp Vault SSH Certificate Signing
vault write ssh/sign/my-role \
    public_key=@~/.ssh/id_ed25519.pub
# Returns signed certificate

2. Commercial SSH Key Management

• SSH.com Universal SSH Key Manager
• Venafi SSH Protect
• Userify
• Teleport

3. Cloud Provider Solutions

• AWS Systems Manager Session Manager (replaces SSH)
• Google OS Login (IAM-based SSH)
• Azure AD SSH authentication

4. Bastion-as-a-Service

• Boundary (HashiCorp)
• Teleport
• StrongDM

These solutions provide:

• Centralized authentication (SSO integration)
• Session recording and audit
• Just-in-time access provisioning
• Automatic credential rotation

We've explored SSH key management comprehensively—from generating individual keys to enterprise-scale certificate infrastructure. Let's consolidate the essential practices:

Module Complete:

With this final page, you've completed the comprehensive SSH module. You now understand SSH from first principles—secure remote access needs, protocol architecture, authentication mechanisms, tunneling capabilities, and key management practices. This knowledge enables you to implement, secure, and manage SSH infrastructure at any scale.