GitOps Principles - Gitops | OneNoughtOne

0/273

System Design (HLD)GitOps

GitOps: Infrastructure and Application Delivery via Git

LevelIntermediate

Duration90 mins

TopicGitOps

1 / 5

GitOps Principles

The Evolution of Infrastructure Delivery

For decades, infrastructure and application deployments were executed through manual processes, custom scripts, and ad-hoc procedures. Operators would SSH into servers, run commands, modify configuration files, and hope that their changes produced the desired outcome. This approach—often called ClickOps or imperative operations—introduced countless failure modes: undocumented changes, configuration drift, inconsistent environments, and the terrifying "works on my machine" phenomenon.

GitOps emerged as a radical rethinking of this paradigm. Rather than treating infrastructure as something to be manipulated through direct commands, GitOps treats the entire system—infrastructure, networking, applications, and configurations—as code stored in Git. The revolutionary insight: if everything is in Git, then Git itself becomes the control plane for all operations.

What You Will Learn

By the end of this page, you will understand the four core principles of GitOps, why Git serves as the ideal foundation for infrastructure management, and how declarative systems combined with automated reconciliation create self-healing, auditable infrastructure that scales from startups to planet-scale enterprises.

The Birth of GitOps

GitOps was formally coined by Weaveworks in 2017, though its principles drew from years of DevOps evolution, infrastructure-as-code practices, and the Kubernetes revolution. The term emerged from observing how cloud-native teams at companies like Google, Netflix, and Spotify were managing increasingly complex distributed systems.

The insight was simple yet profound: Git was already solving most of the hard problems that plagued operations teams. Version control, audit trails, collaboration, rollback capabilities, branching for experimentation—all of these were battle-tested in software development. Why not extend the same model to infrastructure?

The Traditional Approach (Push-Based Deployment):

In traditional CI/CD, a pipeline reacts to code changes, builds artifacts, and then pushes those artifacts to production environments. This model has fundamental weaknesses:

The CI/CD system requires credentials to access production (security risk)
The system state can diverge from the declared state between deployments (configuration drift)
Failed deployments leave systems in unknown states
No single source of truth—state is distributed across the CI system, production, and possibly documentation

The Push Model's Hidden Danger

In push-based deployment, your CI/CD system becomes a powerful attack vector. It holds credentials to your production environment, making it a prime target. Compromise the CI system, and attackers gain the keys to your kingdom. GitOps eliminates this by inverting the model: production pulls changes from Git, never receiving credentials from an external system.

The GitOps Approach (Pull-Based Reconciliation):

GitOps inverts the deployment model. Instead of external systems pushing changes into production, an operator running inside the cluster continuously observes Git and ensures the live system matches the desired state declared in Git.

This inversion has profound implications:

Credentials stay local: The cluster pulls from Git; no external system needs production credentials
Continuous reconciliation: The system perpetually converges toward the desired state
Self-healing infrastructure: Manual changes are automatically reverted
Complete audit trail: Every change is a Git commit with author, timestamp, and message
Simple rollback: Reverting infrastructure is as simple as git revert

Push-Based vs. Pull-Based Deployment Models
Aspect	Traditional (Push)	GitOps (Pull)
Credential management	CI system holds production credentials	Cluster pulls from Git; credentials stay local
Drift detection	Only at deployment time	Continuous; every reconciliation interval
Recovery from manual changes	Must wait for next deployment	Automatic reversion to desired state
Audit trail	Scattered across CI logs	Complete in Git history
Rollback mechanism	Re-run old pipeline (may fail)	Simple git revert
Attack surface	CI system is high-value target	Git is the only entry point

The Four Core Principles of GitOps

The GitOps methodology rests on four foundational principles, formally defined by the OpenGitOps project (a CNCF sandbox project). Understanding these principles deeply—not just superficially—is essential for implementing GitOps correctly and reaping its full benefits.

The Four GitOps Principles

•Declarative Configuration — The entire system is described declaratively. You define what the system should look like, not how to get there. The system's desired state is expressed in a format that can be versioned, audited, and applied automatically.
•Version Controlled, Immutable, and Single Source of Truth — All desired state is stored in Git, making Git the single source of truth. Every change is versioned, creating an immutable audit trail. No change enters the system except through Git.
•Applied Automatically — Software agents automatically apply the desired state from Git to the live system. There is no manual kubectl apply, no SSH to servers, no manual intervention. Approved changes in Git are automatically propagated.
•Continuous Reconciliation — Software agents continuously observe the live system and compare it against the desired state in Git. When drift is detected, agents automatically correct it. The system is self-healing by design.

Principle 1: Declarative Configuration

Declarative configuration means describing the end state of your system, not the steps to achieve it. Consider the difference:

Imperative: "Run apt-get update, then install nginx, then copy this config file to /etc/nginx/, then start the nginx service."

Declarative: "There should be an nginx pod running version 1.24, with this ConfigMap mounted at /etc/nginx/, and the service should expose port 80."

The declarative approach has critical advantages:

Idempotency: Applying the same configuration multiple times produces the same result
Convergence: The system naturally converges toward the desired state
Simplicity: Operators don't need to know the current state to apply changes
Portability: The same declaration works across environments

declarative-vs-imperative.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
# Declarative: Describe the desired end state
# The system figures out how to achieve it
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.24
        ports:
        - containerPort: 80
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "200m"
        volumeMounts:
        - name: nginx-config
          mountPath: /etc/nginx/conf.d
      volumes:
      - name: nginx-config
        configMap:
          name: nginx-config
 
# No "steps" - just declare what should exist
# Kubernetes figures out pod scheduling, networking, etc.

Principle 2: Version Controlled, Immutable, Single Source of Truth

Making Git the single source of truth means that the state in Git is canonical. There is no "well, someone made a manual change that overrides the Git config" scenario—any manual change is either committed to Git or reverted by the reconciliation loop.

Immutability is key: you never modify history, you append to it. Each commit represents a complete snapshot of the desired system state. This creates a complete audit trail:

Who made the change (Git author)
When it was made (timestamp)
What changed (diff)
Why it was made (commit message, linked issue)
Who approved it (code review, merge approval)

This audit trail is invaluable for compliance (SOC 2, HIPAA, PCI-DSS), incident investigation, and understanding system evolution.

Git as Compliance Database

For regulated industries, Git's immutable history serves as a compliance database. Every infrastructure change can be traced to a specific developer, code review, and approval workflow. Auditors can verify that all changes followed proper procedures by examining Git history—no separate audit logs required.

Principle 3: Applied Automatically

The "automatically applied" principle eliminates human operators from the deployment loop. Once a change is merged to the designated branch in Git, software agents detect the change and apply it to the target environment. No one runs kubectl apply manually. No one SSHs into servers.

This automation provides:

Consistency: The same process runs every time, eliminating human error
Speed: Changes propagate in minutes without waiting for operator availability
Scalability: One operator can manage thousands of clusters
Repeatability: Deployments are deterministic, not dependent on individual operator knowledge

Principle 4: Continuous Reconciliation

This is the most transformative principle. The GitOps agent doesn't just deploy once—it continuously monitors the live system and ensures it matches the desired state in Git. This happens at regular intervals (e.g., every 3 minutes) or when Git changes are detected.

Continuous reconciliation creates self-healing infrastructure:

If someone manually deletes a pod, the agent recreates it
If a ConfigMap is modified directly, the agent reverts it
If a new node joins the cluster, the agent applies all relevant configurations
If a deployment rollout fails, the agent detects the discrepancy and can alert operators

This model fundamentally changes how operators think about systems. Instead of asking "how do I fix this?", the question becomes "what state should this be in?" You declare the desired state, and the system perpetually converges toward it.

Converting Mermaid diagram...

Why Git Is the Ideal Foundation

Git wasn't designed for infrastructure management, yet it turns out to be nearly perfect for the job. Understanding why Git works so well illuminates the deeper principles of GitOps and helps you design effective GitOps workflows.

Git's Properties for Infrastructure Management

•Distributed & Resilient — Every clone is a complete backup. If your Git server goes down, every developer's laptop has the full history. For infrastructure management, this means your desired state is never lost.
•Immutable History — Git's append-only history with cryptographic hashes ensures no one can silently modify past states. Every change is traceable, making tampering evident and audits straightforward.
•Branching & Merging — Git's branching model enables experimentation without risk. Create a branch, test configuration changes in a staging environment, and merge to production only when validated.
•Pull Requests & Code Review — Modern Git platforms provide code review workflows. Infrastructure changes go through the same rigorous review as application code, catching errors before they reach production.
•Authentication & Authorization — Git platforms implement sophisticated RBAC. Teams can control who can commit to which repositories and branches, enforcing separation of duties for security.
•Webhooks & Events — Git platforms emit events on commits, enabling real-time triggers for GitOps operators. Changes propagate immediately without polling delays.
•Battle-Tested Scale — Git handles millions of files and thousands of branches. The Linux kernel repository has over 1 million commits. Your infrastructure manifests are trivial by comparison.

The Git Mental Model for Infrastructure:

When using GitOps, think of your Git repository as a time machine for your infrastructure:

HEAD represents the current desired state of your infrastructure
Each commit is a snapshot of the complete desired state at a point in time
git log is your complete infrastructure changelog
git diff shows you exactly what changed between any two states
git bisect can help identify which change introduced a problem
git revert instantly rolls back any change
Branches represent different environments or experimental configurations

This mental model transforms infrastructure management from ad-hoc operations to version-controlled engineering.

The Git Workflow = The Deployment Workflow

In GitOps, your Git workflow is your deployment workflow. Feature branches test configurations in development, pull requests gate changes with reviews, merges trigger deployments. This unification means every skill developers have with Git directly applies to infrastructure. There's no separate 'deployment process' to learn.

The Reconciliation Loop: Deep Dive

The reconciliation loop is the engine that powers GitOps. Understanding its mechanics is essential for troubleshooting, optimization, and designing effective GitOps architectures.

The Basic Reconciliation Cycle:

Observe Git: The operator polls Git (or receives webhook notifications) to detect changes
Retrieve Desired State: The operator reads and parses all manifests from the configured Git path
Observe Live State: The operator queries the Kubernetes API to determine current cluster state
Compute Diff: The operator compares desired state against live state
Apply Changes: If differences exist, the operator applies the necessary changes
Wait: The operator waits for the configured interval before repeating

This cycle runs perpetually, ensuring the cluster continuously converges toward the desired state.

reconciliation-pseudocode.ts
TypeScript
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
// Simplified pseudocode for a GitOps reconciliation loop
interface ReconciliationResult {
  inSync: boolean;
  changesApplied: ResourceChange[];
  errors: Error[];
}
 
class GitOpsOperator {
  private gitRepo: GitRepository;
  private kubeClient: KubernetesClient;
  private syncInterval: number = 180_000; // 3 minutes
  
  async runReconciliationLoop(): Promise<void> {
    while (true) {
      try {
        const result = await this.reconcile();
        
        if (result.inSync) {
          console.log('Cluster in sync with Git');
        } else {
          console.log(`Applied ${result.changesApplied.length} changes`);
        }
        
        if (result.errors.length > 0) {
          this.alertOnErrors(result.errors);
        }
        
        this.emitMetrics(result);
        
      } catch (error) {
        this.handleReconciliationError(error);
      }
      
      await this.sleep(this.syncInterval);
    }
  }
  
  async reconcile(): Promise<ReconciliationResult> {
    // Step 1: Pull latest from Git
    await this.gitRepo.pull();
    
    // Step 2: Read and parse desired state
    const desiredState = await this.gitRepo.parseManifests('deploy/');
    
    // Step 3: Query current cluster state
    const liveState = await this.kubeClient.getCurrentState(
      desiredState.namespaces
    );
    
    // Step 4: Compute diff
    const diff = this.computeDiff(desiredState, liveState);
    
    if (diff.isEmpty()) {
      return { inSync: true, changesApplied: [], errors: [] };
    }
    
    // Step 5: Apply changes (create, update, delete)
    const { applied, errors } = await this.applyChanges(diff);
    
    return {
      inSync: errors.length === 0,
      changesApplied: applied,
      errors: errors,
    };
  }
  
  private computeDiff(
    desired: ResourceSet, 
    live: ResourceSet
  ): Diff {
    const toCreate = desired.filter(d => !live.has(d.id));
    const toUpdate = desired.filter(d => {
      const liveResource = live.get(d.id);
      return liveResource && !liveResource.matches(d);
    });
    const toDelete = live.filter(l => 
      !desired.has(l.id) && l.managedByThis()
    );
    
    return new Diff(toCreate, toUpdate, toDelete);
  }
}

Understanding Drift Detection:

Drift occurs when the live state diverges from the desired state in Git. Drift can happen due to:

Manual changes: Someone runs kubectl edit or kubectl delete
Auto-scaling events: HPA scaling pods beyond declared replicas
Failed deployments: A rollout that's stuck or partially completed
Node failures: Resources being rescheduled or lost
External controllers: Other operators modifying resources

GitOps operators detect drift by comparing the actual observed state with the desired declared state. However, not all drift is equal, and operators provide configuration for handling different scenarios:

Auto-heal drift: Automatically revert any changes (strict GitOps)
Alert on drift: Notify operators but don't auto-correct
Ignore certain fields: Allow HPA to modify replica counts without triggering drift

The Drift Dilemma with HPA

Horizontal Pod Autoscalers (HPA) modify replica counts dynamically based on load. If your GitOps operator strictly enforces the replica count from Git, it will fight the HPA. Most operators provide 'ignore differences' configurations to exclude specific fields from drift detection, allowing controllers like HPA to operate without conflicts.

Security Benefits of GitOps

Security is often an afterthought in deployment pipelines, but GitOps bakes security in from the ground up. The pull-based model and Git-centric workflow provide security benefits that are difficult to achieve with traditional approaches.

GitOps Security Advantages

•Reduced Attack Surface — No external system has credentials to your production clusters. The GitOps operator runs inside the cluster and pulls from Git—Git is the only external dependency that needs access.
•Credential Isolation — CI/CD pipelines don't need kubectl access or cloud provider credentials for production. They just push to Git. The operator handles all cluster interaction with locally-scoped credentials.
•Immutable Audit Trail — Every change is a signed Git commit. Attackers cannot modify infrastructure without leaving a trace. Git's cryptographic hashes ensure history integrity.
•Review Gates — All changes must pass pull request reviews. Infrastructure changes get the same scrutiny as application code. Branch protection rules enforce review requirements.
•Blast Radius Containment — If a single developer's credentials are compromised, they can only push to branches they have access to. Protected branch rules prevent direct production commits.
•Easy Rollback — If a malicious change makes it through, reverting is a simple git revert. No need to reconstruct the previous state from memory or logs.
•Principle of Least Privilege — The GitOps operator has exactly the permissions needed to apply manifests, nothing more. It can't modify things outside its configured scope.

Traditional CI/CD Security Risks

•CI runner has production cluster credentials
•Secrets stored in CI system configuration
•Compromised CI = compromised infrastructure
•Audit trail scattered across CI logs
•Hard to enforce review for emergency fixes
•Rollback requires re-running pipelines

GitOps Security Model

•Operator uses in-cluster credentials only
•Secrets managed by sealed secrets or external stores
•Compromised CI cannot directly affect clusters
•Complete audit trail in Git history
•All changes go through Git review workflow
•Rollback is instant git revert

Zero-Trust with GitOps

GitOps aligns well with zero-trust security models. The cluster trusts only signed commits from verified authors. The operator verifies GPG signatures on commits before applying changes. This cryptographic verification means even if Git credentials are stolen, attackers cannot push changes without the private signing key.

When GitOps Works Best (and When It Doesn't)

GitOps is powerful, but it's not universally applicable. Understanding where GitOps excels—and where it struggles—helps you make informed architectural decisions.

GitOps Suitability Analysis
Scenario	GitOps Fit	Notes
Kubernetes-native infrastructure	Excellent	Kubernetes' declarative model is a perfect match for GitOps
Multi-cluster deployments	Excellent	One Git repo can drive many clusters with minimal overhead
Regulated industries (finance, healthcare)	Excellent	Git's audit trail satisfies compliance requirements
Microservices with frequent deploys	Excellent	Automated reconciliation handles high deployment frequency
Traditional VMs with configuration drift	Good	GitOps with Ansible/Terraform can manage VM infrastructure
Stateful systems with manual intervention	Moderate	Database migrations and manual steps don't fit cleanly
Dynamic auto-scaling resources	Moderate	Requires careful exclusion of scale-managed fields
Serverless functions	Moderate	Possible but less established patterns
Legacy systems without APIs	Poor	GitOps needs a control plane to reconcile against
Highly manual, ad-hoc operations	Poor	GitOps requires discipline; cultural fit matters

The Cultural Requirement:

GitOps is as much a cultural shift as a technical one. Teams must:

Embrace code review for infrastructure: No more "quick fixes" directly in production
Trust the reconciliation loop: Resist the urge to use kubectl apply manually
Document everything in Git: All configuration, all environment variables, all secrets (sealed)
Accept eventual consistency: Changes take minutes to propagate, not seconds

Teams that try to adopt GitOps without cultural buy-in often find themselves fighting the system, making "emergency" changes outside Git, and undermining the benefits they sought.

GitOps ≠ No Ops

GitOps doesn't eliminate operations work—it transforms it. Instead of executing deployments, operators now focus on designing reconciliation strategies, managing Git workflows, improving deploy times, and building tooling around the GitOps pipeline. The work is higher-leverage, but it's still work.

Key Takeaways

We've covered the foundational principles of GitOps—the paradigm shift that makes Git the single source of truth for infrastructure and applications. Let's consolidate the essential concepts:

GitOps Principles Summary

•Four Core Principles — Declarative configuration, version-controlled single source of truth, automatic application, and continuous reconciliation. All four are essential for true GitOps.
•Pull vs. Push — GitOps uses a pull-based model where operators inside the cluster pull state from Git, eliminating the need for external systems to have production credentials.
•Git as the Control Plane — All infrastructure changes flow through Git. The Git workflow (branching, PRs, merges) becomes the deployment workflow.
•Self-Healing Infrastructure — Continuous reconciliation automatically corrects drift, whether from manual changes, failures, or external controllers.
•Security by Design — Reduced attack surface, immutable audit trails, credential isolation, and mandatory code review bake security into the delivery process.
•Cultural Shift Required — GitOps demands discipline: no manual changes, everything in Git, trust the reconciliation loop.

What's Next:

Now that you understand the "what" and "why" of GitOps, the next page dives into the two most popular GitOps tools: ArgoCD and Flux. You'll learn their architectures, feature sets, and when to choose one over the other for your infrastructure.

Page Complete

You now understand the foundational principles of GitOps—the paradigm that transforms infrastructure management from ad-hoc commands to version-controlled, declarative, self-healing systems. Next, we'll explore ArgoCD and Flux, the leading tools that implement these principles.