System Design (HLD)Container Fundamentals

Container Fundamentals

LevelIntermediate

Duration75 mins

TopicContainer Fundamentals

2 / 5

Docker Basics

The Tool That Popularized Containers

While container technologies existed before Docker (LXC, OpenVZ, Solaris Zones), Docker transformed containerization from a systems engineering curiosity into a mainstream development practice. Launched in 2013, Docker provided the user experience layer that made containers accessible to every developer.

Docker's genius wasn't in inventing containers—it was in making them usable. The Dockerfile format, the layered image system, the Docker Hub registry, and the simple CLI combined to create a workflow that felt natural to developers accustomed to version control and package managers.

Today, Docker Desktop has millions of active users, and Docker's image format became the de facto standard (later formalized as OCI). Understanding Docker is essential for any engineer working with modern infrastructure.

What You Will Learn

By the end of this page, you will understand Docker's architecture, master essential Docker commands, learn to write effective Dockerfiles, understand multi-stage builds, and develop best practices for containerized development workflows. You'll have the practical skills to build, run, and ship containerized applications.

Docker Architecture: Client-Server Model

Docker uses a client-server architecture. Understanding this architecture helps you diagnose issues and configure Docker for different environments.

Core Components:

Docker Architecture Components
Component	Role	Details
Docker Client (docker)	User interface	CLI that sends commands to the daemon. Can connect to local or remote daemons.
Docker Daemon (dockerd)	Background service	Listens for API requests, manages images, containers, networks, and volumes.
containerd	Container runtime	Manages container lifecycle (create, start, stop). Industry-standard runtime.
runc	OCI runtime	Low-level tool that creates namespaces and cgroups. Actually runs containers.
Docker Registry	Image storage	Stores and distributes Docker images. Docker Hub is the default public registry.

Docker Architecture Diagram

Architecture

┌───────────────────────────────────────────────────────────────────┐
│                        DOCKER HOST                                 │
│                                                                    │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │                    Docker Daemon (dockerd)                    │ │
│  │                                                               │ │
│  │   ┌─────────────────────────────────────────────────────┐   │ │
│  │   │                    containerd                         │   │ │
│  │   │   ┌─────────┐   ┌─────────┐   ┌─────────┐         │   │ │
│  │   │   │Container│   │Container│   │Container│         │   │ │
│  │   │   │   A     │   │   B     │   │   C     │         │   │ │
│  │   │   │ (runc)  │   │ (runc)  │   │ (runc)  │         │   │ │
│  │   │   └─────────┘   └─────────┘   └─────────┘         │   │ │
│  │   └─────────────────────────────────────────────────────┘   │ │
│  │                                                               │ │
│  │   ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │ │
│  │   │   Images    │  │   Volumes   │  │  Networks   │        │ │
│  │   └─────────────┘  └─────────────┘  └─────────────┘        │ │
│  └──────────────────────────────────────────────────────────────┘ │
│              ▲                                                     │
│              │ REST API (unix socket or TCP)                      │
│              ▼                                                     │
│  ┌────────────────────┐         ┌─────────────────────────────┐  │
│  │  Docker CLI Client │◀───────▶│     Docker Compose          │  │
│  │     (docker)       │         │   (docker-compose / docker) │  │
│  └────────────────────┘         └─────────────────────────────┘  │
└───────────────────────────────────────────────────────────────────┘
              ▲
              │ Push / Pull
              ▼
┌───────────────────────────────────────────────────────────────────┐
│                      Docker Registry                               │
│         (Docker Hub, ECR, GCR, ACR, Harbor, etc.)                 │
└───────────────────────────────────────────────────────────────────┘

Key architectural insights:

Separation of concerns: The CLI doesn't run containers; it sends commands to the daemon
Remote access: Docker clients can connect to remote daemons over TCP (secured with TLS)
containerd independence: containerd can run without dockerd, which Kubernetes exploits directly
Registry integration: Docker daemon handles all registry communication (authentication, layer management)

Docker Desktop vs Docker Engine

Docker Desktop (for Windows/Mac) includes Docker Engine plus additional tooling: a VM to run the Linux kernel, a GUI, Kubernetes integration, and file system syncing. On Linux, Docker Engine runs natively without a VM. Docker Desktop is free for personal and small business use but requires a paid subscription for larger organizations.

Essential Docker Commands

Mastering Docker begins with its command-line interface. The Docker CLI follows a consistent pattern: docker <object> <command> [options]. Let's cover the commands you'll use daily.

Container Lifecycle Commands:

Container Lifecycle
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# Pull an image from registry
docker pull nginx:1.25
 
# Run a container (pull if needed)
docker run nginx:1.25
 
# Run interactively with terminal
docker run -it ubuntu:22.04 /bin/bash
 
# Run detached (background) with port mapping and name
docker run -d --name webserver -p 8080:80 nginx:1.25
 
# List running containers
docker ps
 
# List all containers (including stopped)
docker ps -a
 
# Stop a running container (SIGTERM, then SIGKILL after 10s)
docker stop webserver
 
# Start a stopped container
docker start webserver
 
# Restart a container
docker restart webserver
 
# Remove a stopped container
docker rm webserver
 
# Force remove a running container
docker rm -f webserver
 
# Remove all stopped containers
docker container prune

Image Management Commands:

Image Management
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# List local images
docker images
 
# Pull specific version
docker pull python:3.11-slim
 
# Build image from Dockerfile in current directory
docker build -t myapp:1.0 .
 
# Build with specific Dockerfile
docker build -f Dockerfile.prod -t myapp:prod .
 
# Tag an image
docker tag myapp:1.0 myregistry.com/myapp:1.0
 
# Push to registry
docker push myregistry.com/myapp:1.0
 
# Remove an image
docker rmi myapp:1.0
 
# Remove unused images
docker image prune
 
# Remove ALL unused images (not just dangling)
docker image prune -a
 
# Inspect image layers and metadata
docker image inspect nginx:1.25
 
# View image history (layers)
docker history nginx:1.25

Debugging and Inspection Commands:

Debugging Commands
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# View container logs
docker logs webserver
 
# Follow logs in real-time
docker logs -f webserver
 
# Show last 100 lines with timestamps
docker logs --tail 100 -t webserver
 
# Execute command in running container
docker exec webserver cat /etc/nginx/nginx.conf
 
# Get interactive shell in running container
docker exec -it webserver /bin/sh
 
# View container resource usage
docker stats
 
# View container processes
docker top webserver
 
# Inspect container details (JSON output)
docker inspect webserver
 
# Copy files from container to host
docker cp webserver:/etc/nginx/nginx.conf ./nginx.conf
 
# Copy files from host to container
docker cp ./custom.conf webserver:/etc/nginx/conf.d/

Command Line Shortcuts

Many Docker commands have shortened forms. 'docker container ls' is the same as 'docker ps'. 'docker image ls' is the same as 'docker images'. Container IDs can be abbreviated—you only need enough characters to be unique (often 3-4 characters). Use tab completion by installing Docker's shell completions.

Understanding docker run Flags

The docker run command has dozens of flags that control container behavior. Understanding the most important flags is crucial for both development and production.

Essential docker run Flags:

Critical docker run Flags
Flag	Purpose	Example
-d, --detach	Run in background	docker run -d nginx
-p, --publish	Map container port to host	docker run -p 8080:80 nginx
--name	Assign container name	docker run --name web nginx
-e, --env	Set environment variable	docker run -e DB_HOST=localhost app
--env-file	Read env vars from file	docker run --env-file .env app
-v, --volume	Mount volume or bind mount	docker run -v data:/app/data app
--mount	Mount with explicit options	docker run --mount type=bind,src=.,dst=/app app
-w, --workdir	Set working directory	docker run -w /app node npm start
--network	Connect to network	docker run --network mynet app
--restart	Restart policy	docker run --restart unless-stopped app
--rm	Remove container on exit	docker run --rm ubuntu cat /etc/os-release
-it	Interactive tty (combined)	docker run -it ubuntu bash

Resource Constraint Flags:

Resource Constraints
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Memory limits
docker run --memory=512m myapp          # Hard limit at 512 MB
docker run --memory=512m --memory-swap=1g myapp  # 512 MB RAM + 512 MB swap
 
# CPU limits
docker run --cpus=2 myapp               # Limit to 2 CPU cores
docker run --cpu-shares=512 myapp       # Relative weight (default 1024)
docker run --cpuset-cpus="0,1" myapp    # Pin to specific cores
 
# Combined production example
docker run -d \
  --name myapp \
  --memory=2g \
  --memory-swap=2g \
  --cpus=1.5 \
  --restart=unless-stopped \
  --health-cmd="curl -f http://localhost/health || exit 1" \
  --health-interval=30s \
  --health-timeout=10s \
  --health-retries=3 \
  -p 8080:8080 \
  myapp:latest

Memory Limits and OOM Killer

When a container exceeds its memory limit, the Linux OOM (Out-of-Memory) Killer terminates it. Docker reports this as exit code 137. Always set memory limits in production to prevent one container from consuming all host memory. Use --memory-swap equal to --memory to disable swap and make behavior predictable.

Security-Related Flags:

Security Flags
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Run as specific user (avoid running as root)
docker run --user 1000:1000 myapp
 
# Read-only filesystem
docker run --read-only myapp
 
# Drop all capabilities, add only needed ones
docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE myapp
 
# Disable privilege escalation
docker run --security-opt=no-new-privileges:true myapp
 
# Custom seccomp profile
docker run --security-opt seccomp=/path/to/profile.json myapp
 
# Limit processes (fork bomb protection)
docker run --pids-limit=100 myapp
 
# Never run in production: gives container full host access
# docker run --privileged myapp  # DANGEROUS

Writing Effective Dockerfiles

A Dockerfile is a text file containing instructions to build a Docker image. Each instruction creates a layer in the image. Understanding how to write efficient Dockerfiles is fundamental to containerization.

Dockerfile Instruction Reference:

Core Dockerfile Instructions
Instruction	Purpose	Example
FROM	Base image to start from	FROM python:3.11-slim
WORKDIR	Set working directory	WORKDIR /app
COPY	Copy files from host to image	COPY . /app
RUN	Execute command during build	RUN pip install -r requirements.txt
ENV	Set environment variable	ENV NODE_ENV=production
ARG	Build-time variable	ARG VERSION=1.0
EXPOSE	Document exposed ports (no publish)	EXPOSE 8080
USER	Set user for subsequent commands	USER appuser
CMD	Default command when container starts	CMD ["python", "app.py"]
ENTRYPOINT	Fixed command, CMD becomes args	ENTRYPOINT ["./entrypoint.sh"]
HEALTHCHECK	Container health check	HEALTHCHECK CMD curl -f http://localhost/
VOLUME	Declare volume mount point	VOLUME /data

Example: Well-Structured Python Dockerfile:

Dockerfile
Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Use specific version for reproducibility
FROM python:3.11-slim-bookworm
 
# Set build-time variables
ARG APP_VERSION=1.0.0
 
# Prevent Python from writing pyc files and buffering stdout/stderr
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    APP_VERSION=${APP_VERSION}
 
# Set working directory
WORKDIR /app
 
# Create non-root user for security
RUN groupadd --gid 1000 appgroup && \
    useradd --uid 1000 --gid 1000 --shell /bin/bash appuser
 
# Install dependencies first (for better layer caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
 
# Copy application code
COPY --chown=appuser:appgroup . .
 
# Switch to non-root user
USER appuser
 
# Document the port (doesn't publish it)
EXPOSE 8000
 
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8000/health || exit 1
 
# Default command
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "--workers", "4", "app:app"]

CMD vs ENTRYPOINT

ENTRYPOINT defines the executable that always runs. CMD provides default arguments that can be overridden. Use ENTRYPOINT when your container should always run a specific command (like a web server). Use CMD alone for development containers that might need different commands. Combine them when you have a fixed command with configurable arguments.

Layer Caching and Build Optimization

Docker's layer caching is the key to fast builds. Each Dockerfile instruction creates a layer that can be cached. Understanding caching behavior helps you write Dockerfiles that build in seconds instead of minutes.

How Layer Caching Works:

Docker executes instructions in order
For each instruction, Docker checks if a cached layer exists
Cache hit: Skip instruction, use cached layer
Cache miss: Execute instruction and all subsequent instructions

The critical rule: A cache miss invalidates all subsequent layers.

Poor Caching

•COPY . . (entire source first)
•RUN pip install -r requirements.txt
•Any code change invalidates dependency install
•Rebuild downloads all packages every time

Good Caching

•COPY requirements.txt . (deps first)
•RUN pip install -r requirements.txt
•COPY . . (code after deps)
•Code changes don't invalidate dependencies

Build Optimization Strategies:

Optimization Techniques

•Order by change frequency — Put rarely-changing instructions (base image, system packages) early. Put frequently-changing instructions (application code) late.
•Combine RUN instructions — Each RUN creates a layer. Combine related commands: RUN apt-get update && apt-get install -y pkg1 pkg2 && rm -rf /var/lib/apt/lists/*
•Use .dockerignore — Exclude files from build context that aren't needed in the image (node_modules, .git, tests, docs). Smaller context means faster builds.
•Minimize layers — While caching benefits from granular layers, the final image is smaller with fewer layers. Multi-stage builds help here.
•Choose appropriate base images — Alpine is tiny (~5MB) but uses musl libc which can cause compatibility issues. Slim variants balance size and compatibility.
•Clean up in the same RUN — Files removed in a later layer still exist in earlier layers. Remove temp files in the same RUN command that creates them.

.dockerignore

dockerignore

# Version control
.git
.gitignore
 
# Dependencies (will be installed in container)
node_modules
__pycache__
*.pyc
venv
.venv
 
# Build artifacts
dist
build
*.egg-info
 
# IDE and editor files
.idea
.vscode
*.swp
 
# Tests and docs (unless needed in production)
tests
docs
*.md
!README.md
 
# Local environment
.env
.env.local
*.log
 
# Docker files
Dockerfile*
docker-compose*
.dockerignore

Multi-Stage Builds: Production-Ready Images

Multi-stage builds are one of Docker's most powerful features for creating lean, secure production images. They solve the problem of needing build tools during image creation but not in the final image.

The problem multi-stage builds solve:

Building Go/Rust/Java applications requires compilers, SDKs, and build tools
These tools can add hundreds of megabytes to the final image
Including them increases attack surface and violates least-privilege principles
Before multi-stage builds, teams used complex workarounds (builder pattern)

Multi-Stage Dockerfile (Go)
Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# ===== Stage 1: Build =====
FROM golang:1.21-alpine AS builder
 
# Install build dependencies
RUN apk add --no-cache git ca-certificates
 
WORKDIR /app
 
# Download dependencies (cached layer)
COPY go.mod go.sum ./
RUN go mod download
 
# Build the application
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .
 
# ===== Stage 2: Runtime =====
FROM alpine:3.18 AS runtime
 
# Install runtime dependencies only
RUN apk --no-cache add ca-certificates
 
# Create non-root user
RUN adduser -D -g '' appuser
 
WORKDIR /app
 
# Copy ONLY the binary from builder stage
COPY --from=builder /app/main .
 
# Use non-root user
USER appuser
 
EXPOSE 8080
CMD ["./main"]
 
# Final image: ~20 MB instead of ~1 GB
# No compilers, no source code, minimal attack surface

Multi-stage for Node.js with dependency separation:

Multi-Stage Dockerfile (Node.js)
Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# ===== Stage 1: Dependencies =====
FROM node:20-slim AS deps
 
WORKDIR /app
 
# Install dependencies only
COPY package.json package-lock.json ./
RUN npm ci --only=production
 
# ===== Stage 2: Build =====
FROM node:20-slim AS builder
 
WORKDIR /app
 
COPY package.json package-lock.json ./
RUN npm ci
 
COPY . .
RUN npm run build
 
# ===== Stage 3: Runtime =====
FROM node:20-slim AS runner
 
WORKDIR /app
 
ENV NODE_ENV=production
 
# Create non-root user
RUN addgroup --system --gid 1001 nodejs && \
    adduser --system --uid 1001 nextjs
 
# Copy production dependencies from deps stage
COPY --from=deps /app/node_modules ./node_modules
 
# Copy built application from builder stage
COPY --from=builder --chown=nextjs:nodejs /app/.next ./.next
COPY --from=builder /app/public ./public
COPY --from=builder /app/package.json ./
 
USER nextjs
 
EXPOSE 3000
CMD ["npm", "start"]

Named Stages and COPY --from

You can name stages with 'AS name' and reference them with 'COPY --from=name'. You can also copy from external images: 'COPY --from=nginx:alpine /etc/nginx/nginx.conf /nginx.conf'. This lets you selectively extract artifacts from any image without including it in your build.

Docker Compose: Multi-Container Applications

Real applications rarely run in a single container. A typical web application might include the application server, database, cache, and message queue. Docker Compose defines and runs multi-container applications using a YAML configuration file.

Docker Compose Key Concepts:

Service: A container configuration (image, ports, volumes, environment)
Network: Automatic DNS-based service discovery
Volume: Persistent storage shared across container restarts

docker-compose.yml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
version: '3.8'
 
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "3000:3000"
    environment:
      - DATABASE_URL=postgres://user:pass@db:5432/myapp
      - REDIS_URL=redis://cache:6379
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_started
    volumes:
      - ./src:/app/src  # Development hot reload
    networks:
      - app-network
 
  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
      - POSTGRES_DB=myapp
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user -d myapp"]
      interval: 5s
      timeout: 5s
      retries: 5
    networks:
      - app-network
 
  cache:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    networks:
      - app-network
 
volumes:
  postgres_data:
  redis_data:
 
networks:
  app-network:
    driver: bridge

Essential Docker Compose Commands:

Docker Compose Commands
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Start all services (detached mode)
docker compose up -d
 
# Start and rebuild images
docker compose up -d --build
 
# View logs for all services
docker compose logs -f
 
# View logs for specific service
docker compose logs -f app
 
# Stop all services
docker compose stop
 
# Stop and remove containers, networks
docker compose down
 
# Stop and remove everything including volumes
docker compose down -v
 
# Scale a service
docker compose up -d --scale worker=3
 
# Execute command in running service
docker compose exec app sh
 
# View running services
docker compose ps

Service Discovery in Compose

Containers in the same Compose network can reach each other by service name. In the example above, the app container connects to 'db:5432' and 'cache:6379'. Docker's embedded DNS resolves these names to container IP addresses. No IP addresses are hardcoded.

Summary: Docker Mastery

Let's consolidate our Docker knowledge:

Key Takeaways

•Docker uses a client-server architecture — The CLI talks to the daemon, which manages containers through containerd and runc.
•Master the essential commands — docker run, build, push, pull, exec, logs, and ps will cover 90% of your daily work.
•Understand docker run flags — Port mapping (-p), volumes (-v), environment (-e), user (--user), and resource limits are critical for production.
•Write efficient Dockerfiles — Order instructions by change frequency, combine RUN commands, and use .dockerignore.
•Leverage layer caching — Copy dependencies before source code. A well-structured Dockerfile rebuilds in seconds.
•Use multi-stage builds — Separate build and runtime stages to create minimal, secure production images.
•Docker Compose simplifies multi-container apps — Define services, networks, and volumes in YAML. Service discovery is automatic.
•Always run as non-root — Use USER directive to avoid running containers as root in production.

What's next:

With Docker basics mastered, we'll explore Container Images in depth—understanding how images are structured, versioning strategies, and how to create reproducible, scannable images that form the foundation of your deployment pipeline.

Page Complete

You now have the practical skills to build, run, and manage Docker containers. These fundamentals will serve you whether you're developing locally, deploying to production, or working with container orchestration platforms like Kubernetes.

2 / 5

Loading learning content...

System Design (HLD)Container Fundamentals

Container Fundamentals

LevelIntermediate

Duration75 mins

TopicContainer Fundamentals

2 / 5

Docker Basics

The Tool That Popularized Containers

What You Will Learn

Docker Architecture: Client-Server Model

Docker uses a client-server architecture. Understanding this architecture helps you diagnose issues and configure Docker for different environments.

Core Components:

Docker Architecture Components
Component	Role	Details
Docker Client (docker)	User interface	CLI that sends commands to the daemon. Can connect to local or remote daemons.
Docker Daemon (dockerd)	Background service	Listens for API requests, manages images, containers, networks, and volumes.
containerd	Container runtime	Manages container lifecycle (create, start, stop). Industry-standard runtime.
runc	OCI runtime	Low-level tool that creates namespaces and cgroups. Actually runs containers.
Docker Registry	Image storage	Stores and distributes Docker images. Docker Hub is the default public registry.

Docker Architecture Diagram

Architecture

┌───────────────────────────────────────────────────────────────────┐
│                        DOCKER HOST                                 │
│                                                                    │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │                    Docker Daemon (dockerd)                    │ │
│  │                                                               │ │
│  │   ┌─────────────────────────────────────────────────────┐   │ │
│  │   │                    containerd                         │   │ │
│  │   │   ┌─────────┐   ┌─────────┐   ┌─────────┐         │   │ │
│  │   │   │Container│   │Container│   │Container│         │   │ │
│  │   │   │   A     │   │   B     │   │   C     │         │   │ │
│  │   │   │ (runc)  │   │ (runc)  │   │ (runc)  │         │   │ │
│  │   │   └─────────┘   └─────────┘   └─────────┘         │   │ │
│  │   └─────────────────────────────────────────────────────┘   │ │
│  │                                                               │ │
│  │   ┌─────────────┐  ┌─────────────┐  ┌─────────────┐        │ │
│  │   │   Images    │  │   Volumes   │  │  Networks   │        │ │
│  │   └─────────────┘  └─────────────┘  └─────────────┘        │ │
│  └──────────────────────────────────────────────────────────────┘ │
│              ▲                                                     │
│              │ REST API (unix socket or TCP)                      │
│              ▼                                                     │
│  ┌────────────────────┐         ┌─────────────────────────────┐  │
│  │  Docker CLI Client │◀───────▶│     Docker Compose          │  │
│  │     (docker)       │         │   (docker-compose / docker) │  │
│  └────────────────────┘         └─────────────────────────────┘  │
└───────────────────────────────────────────────────────────────────┘
              ▲
              │ Push / Pull
              ▼
┌───────────────────────────────────────────────────────────────────┐
│                      Docker Registry                               │
│         (Docker Hub, ECR, GCR, ACR, Harbor, etc.)                 │
└───────────────────────────────────────────────────────────────────┘

Key architectural insights:

Separation of concerns: The CLI doesn't run containers; it sends commands to the daemon
Remote access: Docker clients can connect to remote daemons over TCP (secured with TLS)
containerd independence: containerd can run without dockerd, which Kubernetes exploits directly
Registry integration: Docker daemon handles all registry communication (authentication, layer management)

Docker Desktop vs Docker Engine

Essential Docker Commands

Mastering Docker begins with its command-line interface. The Docker CLI follows a consistent pattern: docker <object> <command> [options]. Let's cover the commands you'll use daily.

Container Lifecycle Commands:

Container Lifecycle
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
# Pull an image from registry
docker pull nginx:1.25
 
# Run a container (pull if needed)
docker run nginx:1.25
 
# Run interactively with terminal
docker run -it ubuntu:22.04 /bin/bash
 
# Run detached (background) with port mapping and name
docker run -d --name webserver -p 8080:80 nginx:1.25
 
# List running containers
docker ps
 
# List all containers (including stopped)
docker ps -a
 
# Stop a running container (SIGTERM, then SIGKILL after 10s)
docker stop webserver
 
# Start a stopped container
docker start webserver
 
# Restart a container
docker restart webserver
 
# Remove a stopped container
docker rm webserver
 
# Force remove a running container
docker rm -f webserver
 
# Remove all stopped containers
docker container prune

Image Management Commands:

Image Management
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
# List local images
docker images
 
# Pull specific version
docker pull python:3.11-slim
 
# Build image from Dockerfile in current directory
docker build -t myapp:1.0 .
 
# Build with specific Dockerfile
docker build -f Dockerfile.prod -t myapp:prod .
 
# Tag an image
docker tag myapp:1.0 myregistry.com/myapp:1.0
 
# Push to registry
docker push myregistry.com/myapp:1.0
 
# Remove an image
docker rmi myapp:1.0
 
# Remove unused images
docker image prune
 
# Remove ALL unused images (not just dangling)
docker image prune -a
 
# Inspect image layers and metadata
docker image inspect nginx:1.25
 
# View image history (layers)
docker history nginx:1.25

Debugging and Inspection Commands:

Debugging Commands
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# View container logs
docker logs webserver
 
# Follow logs in real-time
docker logs -f webserver
 
# Show last 100 lines with timestamps
docker logs --tail 100 -t webserver
 
# Execute command in running container
docker exec webserver cat /etc/nginx/nginx.conf
 
# Get interactive shell in running container
docker exec -it webserver /bin/sh
 
# View container resource usage
docker stats
 
# View container processes
docker top webserver
 
# Inspect container details (JSON output)
docker inspect webserver
 
# Copy files from container to host
docker cp webserver:/etc/nginx/nginx.conf ./nginx.conf
 
# Copy files from host to container
docker cp ./custom.conf webserver:/etc/nginx/conf.d/

Command Line Shortcuts

Understanding docker run Flags

The docker run command has dozens of flags that control container behavior. Understanding the most important flags is crucial for both development and production.

Essential docker run Flags:

Critical docker run Flags
Flag	Purpose	Example
-d, --detach	Run in background	docker run -d nginx
-p, --publish	Map container port to host	docker run -p 8080:80 nginx
--name	Assign container name	docker run --name web nginx
-e, --env	Set environment variable	docker run -e DB_HOST=localhost app
--env-file	Read env vars from file	docker run --env-file .env app
-v, --volume	Mount volume or bind mount	docker run -v data:/app/data app
--mount	Mount with explicit options	docker run --mount type=bind,src=.,dst=/app app
-w, --workdir	Set working directory	docker run -w /app node npm start
--network	Connect to network	docker run --network mynet app
--restart	Restart policy	docker run --restart unless-stopped app
--rm	Remove container on exit	docker run --rm ubuntu cat /etc/os-release
-it	Interactive tty (combined)	docker run -it ubuntu bash

Resource Constraint Flags:

Resource Constraints
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Memory limits
docker run --memory=512m myapp          # Hard limit at 512 MB
docker run --memory=512m --memory-swap=1g myapp  # 512 MB RAM + 512 MB swap
 
# CPU limits
docker run --cpus=2 myapp               # Limit to 2 CPU cores
docker run --cpu-shares=512 myapp       # Relative weight (default 1024)
docker run --cpuset-cpus="0,1" myapp    # Pin to specific cores
 
# Combined production example
docker run -d \
  --name myapp \
  --memory=2g \
  --memory-swap=2g \
  --cpus=1.5 \
  --restart=unless-stopped \
  --health-cmd="curl -f http://localhost/health || exit 1" \
  --health-interval=30s \
  --health-timeout=10s \
  --health-retries=3 \
  -p 8080:8080 \
  myapp:latest

Memory Limits and OOM Killer

Security-Related Flags:

Security Flags
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# Run as specific user (avoid running as root)
docker run --user 1000:1000 myapp
 
# Read-only filesystem
docker run --read-only myapp
 
# Drop all capabilities, add only needed ones
docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE myapp
 
# Disable privilege escalation
docker run --security-opt=no-new-privileges:true myapp
 
# Custom seccomp profile
docker run --security-opt seccomp=/path/to/profile.json myapp
 
# Limit processes (fork bomb protection)
docker run --pids-limit=100 myapp
 
# Never run in production: gives container full host access
# docker run --privileged myapp  # DANGEROUS

Writing Effective Dockerfiles

Dockerfile Instruction Reference:

Core Dockerfile Instructions
Instruction	Purpose	Example
FROM	Base image to start from	FROM python:3.11-slim
WORKDIR	Set working directory	WORKDIR /app
COPY	Copy files from host to image	COPY . /app
RUN	Execute command during build	RUN pip install -r requirements.txt
ENV	Set environment variable	ENV NODE_ENV=production
ARG	Build-time variable	ARG VERSION=1.0
EXPOSE	Document exposed ports (no publish)	EXPOSE 8080
USER	Set user for subsequent commands	USER appuser
CMD	Default command when container starts	CMD ["python", "app.py"]
ENTRYPOINT	Fixed command, CMD becomes args	ENTRYPOINT ["./entrypoint.sh"]
HEALTHCHECK	Container health check	HEALTHCHECK CMD curl -f http://localhost/
VOLUME	Declare volume mount point	VOLUME /data

Example: Well-Structured Python Dockerfile:

Dockerfile
Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
# Use specific version for reproducibility
FROM python:3.11-slim-bookworm
 
# Set build-time variables
ARG APP_VERSION=1.0.0
 
# Prevent Python from writing pyc files and buffering stdout/stderr
ENV PYTHONDONTWRITEBYTECODE=1 \
    PYTHONUNBUFFERED=1 \
    APP_VERSION=${APP_VERSION}
 
# Set working directory
WORKDIR /app
 
# Create non-root user for security
RUN groupadd --gid 1000 appgroup && \
    useradd --uid 1000 --gid 1000 --shell /bin/bash appuser
 
# Install dependencies first (for better layer caching)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
 
# Copy application code
COPY --chown=appuser:appgroup . .
 
# Switch to non-root user
USER appuser
 
# Document the port (doesn't publish it)
EXPOSE 8000
 
# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:8000/health || exit 1
 
# Default command
CMD ["gunicorn", "--bind", "0.0.0.0:8000", "--workers", "4", "app:app"]

CMD vs ENTRYPOINT

Layer Caching and Build Optimization

How Layer Caching Works:

Docker executes instructions in order
For each instruction, Docker checks if a cached layer exists
Cache hit: Skip instruction, use cached layer
Cache miss: Execute instruction and all subsequent instructions

The critical rule: A cache miss invalidates all subsequent layers.

Poor Caching

•COPY . . (entire source first)
•RUN pip install -r requirements.txt
•Any code change invalidates dependency install
•Rebuild downloads all packages every time

Good Caching

•COPY requirements.txt . (deps first)
•RUN pip install -r requirements.txt
•COPY . . (code after deps)
•Code changes don't invalidate dependencies

Build Optimization Strategies:

Optimization Techniques

•Order by change frequency — Put rarely-changing instructions (base image, system packages) early. Put frequently-changing instructions (application code) late.
•Combine RUN instructions — Each RUN creates a layer. Combine related commands: RUN apt-get update && apt-get install -y pkg1 pkg2 && rm -rf /var/lib/apt/lists/*
•Use .dockerignore — Exclude files from build context that aren't needed in the image (node_modules, .git, tests, docs). Smaller context means faster builds.
•Minimize layers — While caching benefits from granular layers, the final image is smaller with fewer layers. Multi-stage builds help here.
•Choose appropriate base images — Alpine is tiny (~5MB) but uses musl libc which can cause compatibility issues. Slim variants balance size and compatibility.
•Clean up in the same RUN — Files removed in a later layer still exist in earlier layers. Remove temp files in the same RUN command that creates them.

.dockerignore

dockerignore

# Version control
.git
.gitignore
 
# Dependencies (will be installed in container)
node_modules
__pycache__
*.pyc
venv
.venv
 
# Build artifacts
dist
build
*.egg-info
 
# IDE and editor files
.idea
.vscode
*.swp
 
# Tests and docs (unless needed in production)
tests
docs
*.md
!README.md
 
# Local environment
.env
.env.local
*.log
 
# Docker files
Dockerfile*
docker-compose*
.dockerignore

Multi-Stage Builds: Production-Ready Images

The problem multi-stage builds solve:

Building Go/Rust/Java applications requires compilers, SDKs, and build tools
These tools can add hundreds of megabytes to the final image
Including them increases attack surface and violates least-privilege principles
Before multi-stage builds, teams used complex workarounds (builder pattern)

Multi-Stage Dockerfile (Go)
Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
# ===== Stage 1: Build =====
FROM golang:1.21-alpine AS builder
 
# Install build dependencies
RUN apk add --no-cache git ca-certificates
 
WORKDIR /app
 
# Download dependencies (cached layer)
COPY go.mod go.sum ./
RUN go mod download
 
# Build the application
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .
 
# ===== Stage 2: Runtime =====
FROM alpine:3.18 AS runtime
 
# Install runtime dependencies only
RUN apk --no-cache add ca-certificates
 
# Create non-root user
RUN adduser -D -g '' appuser
 
WORKDIR /app
 
# Copy ONLY the binary from builder stage
COPY --from=builder /app/main .
 
# Use non-root user
USER appuser
 
EXPOSE 8080
CMD ["./main"]
 
# Final image: ~20 MB instead of ~1 GB
# No compilers, no source code, minimal attack surface

Multi-stage for Node.js with dependency separation:

Multi-Stage Dockerfile (Node.js)
Dockerfile
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
# ===== Stage 1: Dependencies =====
FROM node:20-slim AS deps
 
WORKDIR /app
 
# Install dependencies only
COPY package.json package-lock.json ./
RUN npm ci --only=production
 
# ===== Stage 2: Build =====
FROM node:20-slim AS builder
 
WORKDIR /app
 
COPY package.json package-lock.json ./
RUN npm ci
 
COPY . .
RUN npm run build
 
# ===== Stage 3: Runtime =====
FROM node:20-slim AS runner
 
WORKDIR /app
 
ENV NODE_ENV=production
 
# Create non-root user
RUN addgroup --system --gid 1001 nodejs && \
    adduser --system --uid 1001 nextjs
 
# Copy production dependencies from deps stage
COPY --from=deps /app/node_modules ./node_modules
 
# Copy built application from builder stage
COPY --from=builder --chown=nextjs:nodejs /app/.next ./.next
COPY --from=builder /app/public ./public
COPY --from=builder /app/package.json ./
 
USER nextjs
 
EXPOSE 3000
CMD ["npm", "start"]

Named Stages and COPY --from

Docker Compose: Multi-Container Applications

Docker Compose Key Concepts:

Service: A container configuration (image, ports, volumes, environment)
Network: Automatic DNS-based service discovery
Volume: Persistent storage shared across container restarts

docker-compose.yml
YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
version: '3.8'
 
services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "3000:3000"
    environment:
      - DATABASE_URL=postgres://user:pass@db:5432/myapp
      - REDIS_URL=redis://cache:6379
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_started
    volumes:
      - ./src:/app/src  # Development hot reload
    networks:
      - app-network
 
  db:
    image: postgres:15-alpine
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD=pass
      - POSTGRES_DB=myapp
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U user -d myapp"]
      interval: 5s
      timeout: 5s
      retries: 5
    networks:
      - app-network
 
  cache:
    image: redis:7-alpine
    volumes:
      - redis_data:/data
    networks:
      - app-network
 
volumes:
  postgres_data:
  redis_data:
 
networks:
  app-network:
    driver: bridge

Essential Docker Compose Commands:

Docker Compose Commands
Bash
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# Start all services (detached mode)
docker compose up -d
 
# Start and rebuild images
docker compose up -d --build
 
# View logs for all services
docker compose logs -f
 
# View logs for specific service
docker compose logs -f app
 
# Stop all services
docker compose stop
 
# Stop and remove containers, networks
docker compose down
 
# Stop and remove everything including volumes
docker compose down -v
 
# Scale a service
docker compose up -d --scale worker=3
 
# Execute command in running service
docker compose exec app sh
 
# View running services
docker compose ps

Service Discovery in Compose

Summary: Docker Mastery

Let's consolidate our Docker knowledge:

Key Takeaways

•Docker uses a client-server architecture — The CLI talks to the daemon, which manages containers through containerd and runc.
•Master the essential commands — docker run, build, push, pull, exec, logs, and ps will cover 90% of your daily work.
•Understand docker run flags — Port mapping (-p), volumes (-v), environment (-e), user (--user), and resource limits are critical for production.
•Write efficient Dockerfiles — Order instructions by change frequency, combine RUN commands, and use .dockerignore.
•Leverage layer caching — Copy dependencies before source code. A well-structured Dockerfile rebuilds in seconds.
•Use multi-stage builds — Separate build and runtime stages to create minimal, secure production images.
•Docker Compose simplifies multi-container apps — Define services, networks, and volumes in YAML. Service discovery is automatic.
•Always run as non-root — Use USER directive to avoid running containers as root in production.

What's next:

Page Complete

2 / 5