System Design (HLD)CI/CD for Infrastructure

CI/CD for Infrastructure

LevelAdvanced

Duration90 mins

TopicCI/CD for Infrastructure

2 / 5

GitOps Principles

Git as the Single Source of Truth

Imagine a world where every infrastructure state is version-controlled, every change is auditable, and the actual state of your systems is automatically reconciled to match what's declared in your repository. This is the promise of GitOps—a set of practices that fundamentally transforms how organizations manage infrastructure.

GitOps takes the familiar Git-centric workflow that developers already use for application code and extends it to infrastructure and operations. The result is a unified model where Git repositories become the authoritative source for both application deployments and infrastructure configuration, and automated systems continuously ensure that reality matches the declared intent.

What You Will Learn

By the end of this page, you will understand the four core principles of GitOps, how they differ from traditional CI/CD approaches, the architectural patterns that enable GitOps at scale, and the cultural and organizational shifts required to adopt GitOps successfully. You will be prepared to evaluate whether GitOps is right for your organization and how to implement it effectively.

What Is GitOps

GitOps is an operational framework that applies DevOps best practices—version control, collaboration, compliance, and CI/CD—to infrastructure automation. The term was coined by Weaveworks in 2017, though the underlying principles had been evolving in the industry for years.

The GitOps Definition:

GitOps is characterized by four fundamental principles, each building on the others to create a coherent operational model:

The Four Principles of GitOps

•Declarative Configuration — The entire system is described declaratively. Rather than specifying steps to reach a state, you declare the desired end state, and the system determines how to achieve it.
•Version Controlled and Immutable — The desired state is stored in a Git repository as the single source of truth. Changes are made through commits, creating an immutable audit trail.
•Automatic Application — Approved changes in Git are automatically applied to the system. Software agents pull changes and apply them without manual intervention.
•Continuous Reconciliation — Agents continuously monitor actual state versus desired state and correct any drift. The system is self-healing.

Why These Principles Matter:

Each principle addresses a specific operational challenge:

Declarative eliminates procedural complexity and ensures consistency regardless of starting state
Version controlled provides audit trails, rollback capability, and collaboration through pull requests
Automatic application removes human error and delay from deployment
Continuous reconciliation handles drift, partial failures, and ensures long-term consistency

Together, these principles create systems that are predictable, auditable, and resilient.

GitOps Is Not Just CI/CD with Git

A common misconception is that GitOps is simply CI/CD where configuration lives in Git. The key distinction is the reconciliation loop: GitOps systems continuously compare actual state to desired state and correct discrepancies. Traditional CI/CD pipelines are triggered by events and push changes; GitOps agents pull changes and continuously reconcile.

Push-Based vs Pull-Based Models

The distinction between push-based and pull-based deployment models is fundamental to understanding GitOps. While traditional CI/CD uses a push model, GitOps embraces a pull model that offers significant operational advantages.

Push-Based Deployment (Traditional CI/CD):

In a push model, an external CI/CD system detects changes and pushes updates to the target environment. The CI/CD system needs credentials to access the deployment target.

Converting Mermaid diagram...

Pull-Based Deployment (GitOps):

In a pull model, an agent running inside the target environment monitors the Git repository and pulls changes when they occur. The agent only needs read access to Git and write access to its local environment—no external system needs credentials to the deployment target.

Converting Mermaid diagram...

Push vs Pull Deployment Models
Aspect	Push-Based (Traditional)	Pull-Based (GitOps)
Credential Management	CI/CD needs production credentials	Agent runs in-cluster, minimal external exposure
Security Posture	Broader attack surface	Reduced attack surface
Drift Detection	Only at deployment time	Continuous monitoring and correction
Firewall Requirements	Inbound access to target environment	Only outbound access to Git
Recovery Model	Rerun pipeline manually	Automatic reconciliation
Multi-Cluster Scaling	Complex credential distribution	Each cluster pulls independently

Hybrid Approaches

In practice, many organizations use hybrid approaches. CI/CD pipelines handle building, testing, and updating manifests in Git, while GitOps agents handle the actual deployment. This combines the rich tooling of CI/CD with the security and reconciliation benefits of GitOps.

Declarative State Management

The foundation of GitOps is declarative configuration—describing what you want rather than how to achieve it. This paradigm shift has profound implications for how infrastructure is managed, understood, and evolved.

Imperative vs Declarative:

Consider provisioning a load balancer:

imperative.sh
Imperative
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/bin/bash
# Imperative: Specify HOW to create
 
# Check if LB exists
if aws elb describe-load-balancers \
    --names my-app-lb 2>/dev/null; then
  echo "LB exists, checking config..."
  # Compare current config vs desired
  # Update if different
  # Handle various edge cases
else
  echo "Creating load balancer..."
  aws elb create-load-balancer \
    --name my-app-lb \
    --listeners "Protocol=HTTP,..."
fi
 
# Check listener configuration
# Add missing listeners
# Remove extra listeners
# Update health check settings
# ... many more conditionals

declarative.yaml
Declarative
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Declarative: Specify WHAT you want
apiVersion: v1
kind: Service
metadata:
  name: my-app-lb
spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  healthCheckPath: /health
  healthCheckInterval: 30s
  
# The system figures out HOW
# to reach this state from
# any starting point

Why Declarative Is Superior for GitOps:

Declarative configuration provides several key benefits that make it essential for GitOps:

Benefits of Declarative Configuration

•Idempotency — Applying the same configuration multiple times produces the same result, enabling safe retries and reconciliation
•Self-Documenting — The configuration IS the documentation of the desired state; no separate docs to maintain
•Diffable — Changes are clear: what was the previous state, what is the new state, what's the difference
•Rollback Capability — Reverting means applying a previous version of the declaration; Git history provides all previous states
•Convergence — From any starting point, the system converges to the declared state; partial failures are self-healing
•Validation — Declarations can be validated against schemas before application

Not Everything Is Declarative

Some operations are inherently imperative: database migrations, one-time data transformations, secret rotation. GitOps handles these through Job resources or separate management planes. Don't force imperative operations into declarative patterns—recognize the boundaries and design accordingly.

Git as the Source of Truth

In GitOps, Git is not merely a storage location for configuration—it is the authoritative source of truth for the desired state of all systems. This is a strong statement with significant implications.

What 'Source of Truth' Really Means:

If something isn't in Git, it shouldn't exist in the system
If something is different in Git vs. the system, the system is wrong
All changes must flow through Git, never directly to the system
The Git history is the complete audit trail of all changes ever made

Repository Structure Patterns:

Organizations typically use one of several repository structures for GitOps:

GitOps Repository Patterns
Pattern	Description	Best For
Monorepo	All applications and infrastructure in one repository	Small teams, tight coupling, simplified tooling
Environment Repos	Separate repo per environment (dev, staging, prod)	Strong environment isolation, different access controls
App + Config Separation	Application code and deployment config in separate repos	Decoupling deployment from development, config-only changes
Tenant Repos	Separate repos per team or tenant	Multi-tenant platforms, strong team isolation

directory-structure.txt

Tree

# Example GitOps Repository Structure
gitops-config/
├── README.md
├── clusters/
│   ├── production/
│   │   ├── kustomization.yaml
│   │   ├── apps/
│   │   │   ├── frontend/
│   │   │   │   ├── kustomization.yaml
│   │   │   │   ├── deployment.yaml
│   │   │   │   ├── service.yaml
│   │   │   │   └── ingress.yaml
│   │   │   └── backend/
│   │   │       ├── kustomization.yaml
│   │   │       └── ...
│   │   └── infrastructure/
│   │       ├── cert-manager/
│   │       ├── ingress-nginx/
│   │       └── monitoring/
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── apps/
│   └── development/
│       ├── kustomization.yaml
│       └── apps/
├── base/
│   ├── apps/
│   │   ├── frontend/
│   │   │   ├── kustomization.yaml
│   │   │   ├── deployment.yaml
│   │   │   └── service.yaml
│   │   └── backend/
│   └── infrastructure/
└── components/
    ├── observability/
    ├── security/
    └── networking/

Branch Strategies for GitOps:

The branching model affects how changes flow to different environments:

Environment branches — Each environment has its own branch (main→prod, develop→staging). Merges promote changes.
Single branch with paths — One branch (main), environments are directories. Kustomize overlays handle differences.
Trunk-based with tags — Main branch always deployable, tags mark releases, agents subscribe to specific tags.

Most mature GitOps implementations favor the single-branch-with-paths approach, as it simplifies reasoning about state and avoids the complexity of cross-branch merges.

The Env Secrets Problem

Secrets are the main exception to 'everything in Git.' Secrets should NOT be stored in plain text in Git. Solutions include: encrypted secrets (SOPS, sealed-secrets), external secret managers (Vault, AWS Secrets Manager), or operators that inject secrets at runtime (External Secrets Operator).

Continuous Reconciliation

The true power of GitOps emerges from continuous reconciliation—the constant comparison and correction of actual state versus desired state. This is what transforms GitOps from a deployment method into an operational paradigm.

The Reconciliation Loop:

A GitOps agent continuously executes the following cycle:

Converting Mermaid diagram...

What Reconciliation Handles:

Continuous reconciliation automatically addresses scenarios that would require manual intervention in traditional systems:

Scenarios Handled by Reconciliation

•Configuration Drift — Someone manually modified a resource; reconciliation reverts to declared state
•Partial Failures — A deployment partially succeeded; reconciliation completes the remaining changes
•Resource Deletion — A resource was accidentally deleted; reconciliation recreates it
•New Cluster Setup — A fresh cluster receives all configuration automatically
•Secret Rotation — Updated secrets in external stores are automatically propagated
•Dependency Changes — When dependencies change (new cert-manager version), dependent resources update

Reconciliation Configuration Options
Setting	Typical Value	Purpose
Sync Interval	1-5 minutes	How often to check for changes
Retry Interval	1 minute	How often to retry failed syncs
Timeout	10 minutes	Maximum time for a sync operation
Prune	enabled/disabled	Whether to delete resources removed from Git
Self-Heal	enabled/disabled	Whether to revert manual changes

The Prune Decision

Enabling 'prune' means resources deleted from Git will be deleted from the cluster. This is powerful but dangerous if not carefully managed. Many organizations start with prune disabled and enable it only after establishing strong Git workflow discipline.

GitOps Tooling Landscape

Several mature tools implement GitOps patterns, each with different strengths and design philosophies. Understanding the landscape helps in selecting the right tool for your context.

Major GitOps Implementations:

GitOps Tools Comparison
Tool	Approach	Key Strengths	Considerations
ArgoCD	application-centric controller with rich UI	Excellent visualization, application grouping, progressive delivery	More opinionated, requires CRDs for configuration
Flux	toolkit of modular controllers	Highly composable, native Kustomize/Helm support, lightweight	Less visual UI, steeper initial learning curve
Jenkins X	full CI/CD platform with GitOps	Integrated pipeline + GitOps, familiar Jenkins ecosystem	Heavier footprint, more complexity
Rancher Fleet	multi-cluster at scale	Designed for thousands of clusters, GitOps bundles	Best for large fleet management scenarios

argocd-application.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# ArgoCD Application Definition
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-frontend-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  
  source:
    repoURL: https://github.com/org/gitops-config.git
    targetRevision: main
    path: clusters/production/apps/frontend
    
    # Kustomize configuration
    kustomize:
      images:
        - my-registry/frontend:v1.2.3
        
  destination:
    server: https://kubernetes.default.svc
    namespace: frontend
    
  syncPolicy:
    automated:
      prune: true        # Delete resources removed from Git
      selfHeal: true     # Revert manual changes
      allowEmpty: false  # Don't allow empty syncs
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

Tool Selection Guidance

Choose ArgoCD if you value visualization and have teams that prefer GUI-based workflows. Choose Flux if you want maximum flexibility and prefer a 'Kubernetes-native' composition approach. Both are CNCF graduated projects and production-ready.

GitOps for Infrastructure (Not Just Applications)

While GitOps is often associated with Kubernetes application deployments, its principles apply equally to infrastructure management. Extending GitOps beyond applications creates a unified operational model for everything.

Infrastructure GitOps Patterns:

GitOps Applied to Infrastructure

•Cluster Configuration — CNI plugins, storage classes, RBAC policies, all managed through GitOps
•Cloud Resources — Using Crossplane or ACK to manage AWS/Azure/GCP resources declaratively in Kubernetes
•Terraform with GitOps — Tools like Atlantis or tf-controller that apply Terraform through GitOps workflows
•Policy Management — OPA/Gatekeeper policies versioned in Git and reconciled to clusters
•Secret Management — External Secrets Operator syncing from Vault/AWS Secrets Manager based on Git declarations

crossplane-rds.yaml
Crossplane
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Managing AWS RDS via GitOps with Crossplane
apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  name: production-database
  namespace: infrastructure
spec:
  forProvider:
    region: us-west-2
    allocatedStorage: 100
    dbInstanceClass: db.r5.large
    engine: postgres
    engineVersion: "15.4"
    masterUsername: admin
    dbName: production
    publiclyAccessible: false
    storageEncrypted: true
    storageType: gp3
    vpcSecurityGroupIds:
      - sg-0123456789abcdef0
    dbSubnetGroupName: private-subnets
    backupRetentionPeriod: 30
    multiAZ: true
    deletionProtection: true
    
  # Credentials stored in Kubernetes Secret
  writeConnectionSecretToRef:
    namespace: production
    name: rds-connection
    
---
# This RDS instance is now managed by GitOps:
# - Changes in Git trigger reconciliation
# - Drift is automatically detected and corrected
# - Full audit trail in Git history
# - Rollback by reverting commits

The Terraform + GitOps Integration:

For organizations with existing Terraform investments, several tools bridge Terraform and GitOps:

Atlantis — GitHub/GitLab integration for Terraform plan/apply through PRs
Terraform Cloud/Enterprise — HashiCorp's managed GitOps-like workflow
tf-controller — Flux-native Terraform controller running in cluster
Spacelift — SaaS platform with GitOps workflows for Terraform

These tools apply GitOps principles (PR-based workflow, automated application, state tracking) while preserving Terraform's strengths for cloud resource management.

The Kubernetes Control Plane Advantage

Crossplane and similar tools leverage Kubernetes as a universal control plane. Any resource—cloud or on-premises—can be managed through Kubernetes-native APIs. This unifies operations across platforms and enables GitOps for everything, not just containers.

Organizational and Cultural Shifts

Adopting GitOps is as much a cultural transformation as a technical one. The shift from imperative, ad-hoc operations to declarative, Git-centered workflows requires changes in how teams work, communicate, and think about system management.

Key Cultural Shifts:

Traditional Ops Culture

•SSH into servers to diagnose and fix
•Hotfixes applied directly in production
•Documentation separate from configuration
•Tribal knowledge about system state
•Manual rollbacks under pressure
•Operations as gatekeeper

GitOps Culture

•All changes through pull requests
•Production is read-only to humans
•Configuration IS documentation
•Git history is authoritative record
•Rollback = revert commit + sync
•Operations as enabler and reviewer

Skills Development:

GitOps requires new skills across the organization:

Developers — Must understand declarative configuration, Kubernetes manifests, Kustomize/Helm
Operators — Shift from hands-on fixing to observing reconciliation, designing configurations
Security — Focus on Git access controls, secret management, policy-as-code
Platform teams — Build and maintain the GitOps platform itself

Adoption Strategies:

Successful GitOps adoption typically follows this pattern:

Pilot — Start with a non-critical application, prove the workflow
Standardize — Establish patterns, templates, and conventions
Educate — Train teams, document practices, provide support
Migrate — Systematically move workloads to GitOps management
Enforce — Make GitOps the default, restrict direct cluster access

The Emergency Access Question

One common objection: 'What if we need to fix something urgently and can't wait for a PR?' This is valid. GitOps maturity requires fast PR workflows (auto-merge for low-risk changes) and 'break glass' procedures for genuine emergencies. But emergency access should be audited, and any manual changes should trigger alerts and prompt cleanup.

Summary: GitOps Principles

GitOps represents a maturation of infrastructure and deployment practice, applying software engineering discipline to operations. The key principles to remember:

Key Takeaways

•Declarative configuration describes desired state, not procedures—enabling idempotency, diffability, and convergence
•Git is the single source of truth — If it's not in Git, it shouldn't exist; all changes flow through commits
•Pull-based deployment improves security by keeping production credentials in-cluster, not in CI/CD systems
•Continuous reconciliation is what distinguishes GitOps from CI/CD—systems self-heal and converge
•GitOps extends beyond applications to infrastructure, policies, and any declarative configuration
•Tooling choices (ArgoCD, Flux) depend on your needs: visualization, composability, scale
•Cultural transformation is as important as technical implementation—processes and skills must evolve

What's Next:

Now that you understand GitOps principles, the next page explores Pull Request Workflows—the structured processes through which changes enter the Git repository and ultimately flow to your systems.

Page Complete

You now understand the core principles of GitOps, the distinction between push and pull deployment models, and how continuous reconciliation enables self-healing systems. You're prepared to evaluate GitOps adoption for your organization.

2 / 5

Loading learning content...

System Design (HLD)CI/CD for Infrastructure

CI/CD for Infrastructure

LevelAdvanced

Duration90 mins

TopicCI/CD for Infrastructure

2 / 5

GitOps Principles

Git as the Single Source of Truth

What You Will Learn

What Is GitOps

The GitOps Definition:

GitOps is characterized by four fundamental principles, each building on the others to create a coherent operational model:

The Four Principles of GitOps

•Declarative Configuration — The entire system is described declaratively. Rather than specifying steps to reach a state, you declare the desired end state, and the system determines how to achieve it.
•Version Controlled and Immutable — The desired state is stored in a Git repository as the single source of truth. Changes are made through commits, creating an immutable audit trail.
•Automatic Application — Approved changes in Git are automatically applied to the system. Software agents pull changes and apply them without manual intervention.
•Continuous Reconciliation — Agents continuously monitor actual state versus desired state and correct any drift. The system is self-healing.

Why These Principles Matter:

Each principle addresses a specific operational challenge:

Declarative eliminates procedural complexity and ensures consistency regardless of starting state
Version controlled provides audit trails, rollback capability, and collaboration through pull requests
Automatic application removes human error and delay from deployment
Continuous reconciliation handles drift, partial failures, and ensures long-term consistency

Together, these principles create systems that are predictable, auditable, and resilient.

GitOps Is Not Just CI/CD with Git

Push-Based vs Pull-Based Models

Push-Based Deployment (Traditional CI/CD):

In a push model, an external CI/CD system detects changes and pushes updates to the target environment. The CI/CD system needs credentials to access the deployment target.

Converting Mermaid diagram...

Pull-Based Deployment (GitOps):

Converting Mermaid diagram...

Push vs Pull Deployment Models
Aspect	Push-Based (Traditional)	Pull-Based (GitOps)
Credential Management	CI/CD needs production credentials	Agent runs in-cluster, minimal external exposure
Security Posture	Broader attack surface	Reduced attack surface
Drift Detection	Only at deployment time	Continuous monitoring and correction
Firewall Requirements	Inbound access to target environment	Only outbound access to Git
Recovery Model	Rerun pipeline manually	Automatic reconciliation
Multi-Cluster Scaling	Complex credential distribution	Each cluster pulls independently

Hybrid Approaches

Declarative State Management

Imperative vs Declarative:

Consider provisioning a load balancer:

imperative.sh
Imperative
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/bin/bash
# Imperative: Specify HOW to create
 
# Check if LB exists
if aws elb describe-load-balancers \
    --names my-app-lb 2>/dev/null; then
  echo "LB exists, checking config..."
  # Compare current config vs desired
  # Update if different
  # Handle various edge cases
else
  echo "Creating load balancer..."
  aws elb create-load-balancer \
    --name my-app-lb \
    --listeners "Protocol=HTTP,..."
fi
 
# Check listener configuration
# Add missing listeners
# Remove extra listeners
# Update health check settings
# ... many more conditionals

declarative.yaml
Declarative
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# Declarative: Specify WHAT you want
apiVersion: v1
kind: Service
metadata:
  name: my-app-lb
spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  healthCheckPath: /health
  healthCheckInterval: 30s
  
# The system figures out HOW
# to reach this state from
# any starting point

Why Declarative Is Superior for GitOps:

Declarative configuration provides several key benefits that make it essential for GitOps:

Benefits of Declarative Configuration

•Idempotency — Applying the same configuration multiple times produces the same result, enabling safe retries and reconciliation
•Self-Documenting — The configuration IS the documentation of the desired state; no separate docs to maintain
•Diffable — Changes are clear: what was the previous state, what is the new state, what's the difference
•Rollback Capability — Reverting means applying a previous version of the declaration; Git history provides all previous states
•Convergence — From any starting point, the system converges to the declared state; partial failures are self-healing
•Validation — Declarations can be validated against schemas before application

Not Everything Is Declarative

Git as the Source of Truth

What 'Source of Truth' Really Means:

If something isn't in Git, it shouldn't exist in the system
If something is different in Git vs. the system, the system is wrong
All changes must flow through Git, never directly to the system
The Git history is the complete audit trail of all changes ever made

Repository Structure Patterns:

Organizations typically use one of several repository structures for GitOps:

GitOps Repository Patterns
Pattern	Description	Best For
Monorepo	All applications and infrastructure in one repository	Small teams, tight coupling, simplified tooling
Environment Repos	Separate repo per environment (dev, staging, prod)	Strong environment isolation, different access controls
App + Config Separation	Application code and deployment config in separate repos	Decoupling deployment from development, config-only changes
Tenant Repos	Separate repos per team or tenant	Multi-tenant platforms, strong team isolation

directory-structure.txt

Tree

# Example GitOps Repository Structure
gitops-config/
├── README.md
├── clusters/
│   ├── production/
│   │   ├── kustomization.yaml
│   │   ├── apps/
│   │   │   ├── frontend/
│   │   │   │   ├── kustomization.yaml
│   │   │   │   ├── deployment.yaml
│   │   │   │   ├── service.yaml
│   │   │   │   └── ingress.yaml
│   │   │   └── backend/
│   │   │       ├── kustomization.yaml
│   │   │       └── ...
│   │   └── infrastructure/
│   │       ├── cert-manager/
│   │       ├── ingress-nginx/
│   │       └── monitoring/
│   ├── staging/
│   │   ├── kustomization.yaml
│   │   └── apps/
│   └── development/
│       ├── kustomization.yaml
│       └── apps/
├── base/
│   ├── apps/
│   │   ├── frontend/
│   │   │   ├── kustomization.yaml
│   │   │   ├── deployment.yaml
│   │   │   └── service.yaml
│   │   └── backend/
│   └── infrastructure/
└── components/
    ├── observability/
    ├── security/
    └── networking/

Branch Strategies for GitOps:

The branching model affects how changes flow to different environments:

Environment branches — Each environment has its own branch (main→prod, develop→staging). Merges promote changes.
Single branch with paths — One branch (main), environments are directories. Kustomize overlays handle differences.
Trunk-based with tags — Main branch always deployable, tags mark releases, agents subscribe to specific tags.

Most mature GitOps implementations favor the single-branch-with-paths approach, as it simplifies reasoning about state and avoids the complexity of cross-branch merges.

The Env Secrets Problem

Continuous Reconciliation

The Reconciliation Loop:

A GitOps agent continuously executes the following cycle:

Converting Mermaid diagram...

What Reconciliation Handles:

Continuous reconciliation automatically addresses scenarios that would require manual intervention in traditional systems:

Scenarios Handled by Reconciliation

•Configuration Drift — Someone manually modified a resource; reconciliation reverts to declared state
•Partial Failures — A deployment partially succeeded; reconciliation completes the remaining changes
•Resource Deletion — A resource was accidentally deleted; reconciliation recreates it
•New Cluster Setup — A fresh cluster receives all configuration automatically
•Secret Rotation — Updated secrets in external stores are automatically propagated
•Dependency Changes — When dependencies change (new cert-manager version), dependent resources update

Reconciliation Configuration Options
Setting	Typical Value	Purpose
Sync Interval	1-5 minutes	How often to check for changes
Retry Interval	1 minute	How often to retry failed syncs
Timeout	10 minutes	Maximum time for a sync operation
Prune	enabled/disabled	Whether to delete resources removed from Git
Self-Heal	enabled/disabled	Whether to revert manual changes

The Prune Decision

GitOps Tooling Landscape

Several mature tools implement GitOps patterns, each with different strengths and design philosophies. Understanding the landscape helps in selecting the right tool for your context.

Major GitOps Implementations:

GitOps Tools Comparison
Tool	Approach	Key Strengths	Considerations
ArgoCD	application-centric controller with rich UI	Excellent visualization, application grouping, progressive delivery	More opinionated, requires CRDs for configuration
Flux	toolkit of modular controllers	Highly composable, native Kustomize/Helm support, lightweight	Less visual UI, steeper initial learning curve
Jenkins X	full CI/CD platform with GitOps	Integrated pipeline + GitOps, familiar Jenkins ecosystem	Heavier footprint, more complexity
Rancher Fleet	multi-cluster at scale	Designed for thousands of clusters, GitOps bundles	Best for large fleet management scenarios

argocd-application.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# ArgoCD Application Definition
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-frontend-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default
  
  source:
    repoURL: https://github.com/org/gitops-config.git
    targetRevision: main
    path: clusters/production/apps/frontend
    
    # Kustomize configuration
    kustomize:
      images:
        - my-registry/frontend:v1.2.3
        
  destination:
    server: https://kubernetes.default.svc
    namespace: frontend
    
  syncPolicy:
    automated:
      prune: true        # Delete resources removed from Git
      selfHeal: true     # Revert manual changes
      allowEmpty: false  # Don't allow empty syncs
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
    retry:
      limit: 5
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

Tool Selection Guidance

GitOps for Infrastructure (Not Just Applications)

Infrastructure GitOps Patterns:

GitOps Applied to Infrastructure

•Cluster Configuration — CNI plugins, storage classes, RBAC policies, all managed through GitOps
•Cloud Resources — Using Crossplane or ACK to manage AWS/Azure/GCP resources declaratively in Kubernetes
•Terraform with GitOps — Tools like Atlantis or tf-controller that apply Terraform through GitOps workflows
•Policy Management — OPA/Gatekeeper policies versioned in Git and reconciled to clusters
•Secret Management — External Secrets Operator syncing from Vault/AWS Secrets Manager based on Git declarations

crossplane-rds.yaml
Crossplane
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
# Managing AWS RDS via GitOps with Crossplane
apiVersion: database.aws.crossplane.io/v1beta1
kind: RDSInstance
metadata:
  name: production-database
  namespace: infrastructure
spec:
  forProvider:
    region: us-west-2
    allocatedStorage: 100
    dbInstanceClass: db.r5.large
    engine: postgres
    engineVersion: "15.4"
    masterUsername: admin
    dbName: production
    publiclyAccessible: false
    storageEncrypted: true
    storageType: gp3
    vpcSecurityGroupIds:
      - sg-0123456789abcdef0
    dbSubnetGroupName: private-subnets
    backupRetentionPeriod: 30
    multiAZ: true
    deletionProtection: true
    
  # Credentials stored in Kubernetes Secret
  writeConnectionSecretToRef:
    namespace: production
    name: rds-connection
    
---
# This RDS instance is now managed by GitOps:
# - Changes in Git trigger reconciliation
# - Drift is automatically detected and corrected
# - Full audit trail in Git history
# - Rollback by reverting commits

The Terraform + GitOps Integration:

For organizations with existing Terraform investments, several tools bridge Terraform and GitOps:

Atlantis — GitHub/GitLab integration for Terraform plan/apply through PRs
Terraform Cloud/Enterprise — HashiCorp's managed GitOps-like workflow
tf-controller — Flux-native Terraform controller running in cluster
Spacelift — SaaS platform with GitOps workflows for Terraform

These tools apply GitOps principles (PR-based workflow, automated application, state tracking) while preserving Terraform's strengths for cloud resource management.

The Kubernetes Control Plane Advantage

Organizational and Cultural Shifts

Key Cultural Shifts:

Traditional Ops Culture

•SSH into servers to diagnose and fix
•Hotfixes applied directly in production
•Documentation separate from configuration
•Tribal knowledge about system state
•Manual rollbacks under pressure
•Operations as gatekeeper

GitOps Culture

•All changes through pull requests
•Production is read-only to humans
•Configuration IS documentation
•Git history is authoritative record
•Rollback = revert commit + sync
•Operations as enabler and reviewer

Skills Development:

GitOps requires new skills across the organization:

Developers — Must understand declarative configuration, Kubernetes manifests, Kustomize/Helm
Operators — Shift from hands-on fixing to observing reconciliation, designing configurations
Security — Focus on Git access controls, secret management, policy-as-code
Platform teams — Build and maintain the GitOps platform itself

Adoption Strategies:

Successful GitOps adoption typically follows this pattern:

Pilot — Start with a non-critical application, prove the workflow
Standardize — Establish patterns, templates, and conventions
Educate — Train teams, document practices, provide support
Migrate — Systematically move workloads to GitOps management
Enforce — Make GitOps the default, restrict direct cluster access

The Emergency Access Question

Summary: GitOps Principles

GitOps represents a maturation of infrastructure and deployment practice, applying software engineering discipline to operations. The key principles to remember:

Key Takeaways

•Declarative configuration describes desired state, not procedures—enabling idempotency, diffability, and convergence
•Git is the single source of truth — If it's not in Git, it shouldn't exist; all changes flow through commits
•Pull-based deployment improves security by keeping production credentials in-cluster, not in CI/CD systems
•Continuous reconciliation is what distinguishes GitOps from CI/CD—systems self-heal and converge
•GitOps extends beyond applications to infrastructure, policies, and any declarative configuration
•Tooling choices (ArgoCD, Flux) depend on your needs: visualization, composability, scale
•Cultural transformation is as important as technical implementation—processes and skills must evolve

What's Next:

Page Complete

2 / 5