ArgoCD — Deep Dive · k8s-observability-platform

01 GitOps — Git as the Source of Truth Concept

Traditional workflow (imperative)
────────────────────────────────────────────────
Developer → kubectl apply -f manifests/ → Cluster
  ↑ manual, error-prone, no audit trail
  ↑ cluster can drift from what's in Git
  ↑ no rollback without knowing previous state


GitOps workflow (declarative)
────────────────────────────────────────────────
Developer → git push → GitHub repo
                           ↓
                    ArgoCD watches repo  (polls every 3 min or webhook)
                           ↓
                    Detects diff between Git ↔ Cluster
                           ↓
                    Auto-syncs cluster to match Git
                           ↓
                    Cluster always = what's in Git

The key insight: Git becomes the only place you make changes. ArgoCD's job is to make the cluster look exactly like Git — always. If someone manually changes something in the cluster, ArgoCD reverts it (selfHeal: true).

02 Installation Ansible

# roles/argocd/tasks/main.yml

# 1. Create namespace
- name: Create argocd namespace
  command: kubectl create namespace argocd

# 2. Install all ArgoCD components via official manifest
- name: Install ArgoCD
  command: >
    kubectl apply --server-side --force-conflicts -n argocd -f
    https://raw.githubusercontent.com/argoproj/argo-cd/v2.13.3/manifests/install.yaml

Single manifest install — ArgoCD's install.yaml creates everything in one apply: the namespace, RBAC rules, CRDs, and all five components (redis, repo-server, application-controller, dex-server, argocd-server).
--server-side --force-conflicts — same pattern as Calico and MetalLB. ArgoCD CRDs are large enough to exceed the 262 KB client-side annotation limit.
v2.13.3 pinned — set in group_vars/all.yml as argocd_version. Changing it there updates the install URL in one place.

03 Insecure Mode — HTTP behind NGINX Config

# roles/argocd/files/argocd-insecure-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name:      argocd-cmd-params-cm
  namespace: argocd
data:
  server.insecure: "true"   # run HTTP instead of HTTPS

By default, argocd-server terminates its own TLS and serves HTTPS. This creates a problem when placed behind an HTTP NGINX Ingress — NGINX tries to talk HTTP to the backend, but the backend insists on HTTPS, causing a connection error.

server.insecure: true — tells argocd-server to serve plain HTTP. TLS termination becomes NGINX's responsibility (or can be added later with cert-manager).
argocd-cmd-params-cm — the official ArgoCD ConfigMap for runtime flags. ArgoCD reads it on startup, so the server needs a rollout restart after the ConfigMap is applied for the change to take effect.
Ansible handles the restart — after applying the ConfigMap, the argocd role runs kubectl rollout restart deployment/argocd-server and waits for it to be Ready again before proceeding.

Security note: Insecure mode is fine for a local lab. In production you'd use cert-manager to issue a TLS certificate and configure NGINX with TLS termination, keeping the ArgoCD UI on HTTPS.

04 Ingress — argocd.lab.local YAML

# roles/argocd/files/argocd-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name:      argocd-ingress
  namespace: argocd
  annotations:
    # tell NGINX the backend speaks HTTP (not HTTPS)
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  ingressClassName: nginx
  rules:
    - host: argocd.lab.local
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 80

backend-protocol: HTTP — without this annotation, NGINX assumes the backend is HTTP by default but argocd-server was originally HTTPS. Explicitly setting HTTP ensures no protocol mismatch after applying the insecure ConfigMap.
ingressClassName: nginx — selects our NGINX Ingress controller (installed in Phase 02) as the controller to manage this Ingress resource.
host: argocd.lab.local — mapped to 192.168.56.200 in your Windows hosts file. All requests with this Host header are routed to the argocd-server service on port 80.

05 Application CRD — Wiring ArgoCD to GitHub YAML

# roles/online-boutique/files/argocd-app.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name:      online-boutique
  namespace: argocd
spec:
  project: default
  source:
    repoURL:        https://github.com/GoogleCloudPlatform/microservices-demo
    targetRevision: HEAD
    path:           release
    directory:
      include: "kubernetes-manifests.yaml"  # skip istio-manifests.yaml
  destination:
    server:    https://kubernetes.default.svc
    namespace: online-boutique
  syncPolicy:
    automated:
      prune:     true   # delete resources removed from Git
      selfHeal: true   # revert manual cluster changes
    syncOptions:
      - CreateNamespace=true

The Application is ArgoCD's core CRD. It defines the link between a Git source and a cluster destination — ArgoCD continuously reconciles the two.

targetRevision: HEAD — tracks the latest commit on the default branch. We use HEAD instead of a pinned tag because the microservices-demo release/ folder at some tags includes Istio CRDs we don't have. HEAD ensures current k8s API compatibility.
directory.include: kubernetes-manifests.yaml — the release/ folder contains both kubernetes-manifests.yaml and istio-manifests.yaml. Without this filter, ArgoCD tries to apply Istio resources (VirtualService, ServiceEntry, Gateway) that require Istio CRDs which aren't installed — causing a SyncFailed error.
prune: true — if a resource is removed from the Git manifest, ArgoCD deletes it from the cluster. Keeps cluster in sync with Git automatically.
selfHeal: true — if someone manually edits a resource in the cluster (e.g. scales a deployment), ArgoCD reverts it back to what Git says within 3 minutes. Git is always the truth.
CreateNamespace=true — ArgoCD creates the online-boutique namespace automatically before applying resources. Without this, the sync fails if the namespace doesn't exist.

06 Five ArgoCD Components Concept

argocd-server The API server and UI. Handles all client requests — browser, CLI, and API. Runs in insecure HTTP mode in our setup so it works behind NGINX Ingress.

application-controller The reconciliation loop. Compares desired state (Git) vs actual state (cluster) every 3 minutes. When drift is detected, it triggers a sync. Runs as a StatefulSet for stable identity.

repo-server Clones Git repos, renders manifests (plain YAML, Helm, Kustomize), and caches results. Runs separately from the controller so rendering doesn't block syncing.

redis In-memory cache for application state, session data, and rendered manifests. Reduces redundant Git clones and API server calls between components.

07 Verify & Access Commands

# All ArgoCD pods Running?
kubectl get pods -n argocd

# ArgoCD app sync status
kubectl get application online-boutique -n argocd
# Expected: SYNC STATUS=Synced  HEALTH STATUS=Healthy

# Get initial admin password (only exists before first login change)
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

# Browser access
# URL:      http://argocd.lab.local
# Username: admin
# Password: (output from above command)

The ArgoCD UI at http://argocd.lab.local shows every Application, its sync status, health of each resource, and full deployment history. The initial admin password is stored in a Kubernetes secret — it's auto-generated and should be changed after first login.

Tip: In the ArgoCD UI, click the online-boutique app to see a live graph of all 11 microservices, their health status, and resource tree. Any sync errors appear here with full error messages — much easier to debug than running kubectl describe for each resource.