Kubernetes Project

Deploying a Multi-Environment Application with Argo CD🦑and Helm: A Complete GitOps🔄Guide

Hardik Arora — Fri, 21 Nov 2025 18:53:09 GMT

GitOps has evolved from a niche concept to the cornerstone of modern Kubernetes delivery. This comprehensive guide details the deployment and management of a multi-environment microservice (Dev, QA, Prod) using Argo CD and Helm, with a sample Leaderboard application as the focal point. It covers the handling of configuration drift, resolution of production issues, and the implementation of the App of Apps pattern for scalable and declarative management.

📘 What This Guide Covers

How Argo CD Applications are created and managed
Using Helm with environment-specific values
Detecting and fixing configuration drift
Understanding and applying the App of Apps pattern
Resolving real-world production issues
Following GitOps best practices end-to-end

📂 Repository Structure

🔗 GitHub Repository

You can explore the complete implementation of this setup, including Helm charts, environment values, and Argo CD Application manifests here:
GitHub: https://github.com/barbaria888/argo-cd-leaderboard-app

🏅 Author Achievement

This project also contributed to earning Akuity Understanding Gitops with ArgoCD badge

A clear structure is at the heart of good GitOps. The repository is organized into Application manifests and Helm charts:

📦 argo-cd-leaderboard-app/
├── apps/
│   ├── leaderboard-dev.yaml
│   ├── leaderboard-qa.yaml
│   └── leaderboard-prod.yaml
│
└── charts/
    └── leaderboard/
        ├── Chart.yaml
        ├── templates/
        ├── values.yaml
        ├── values-dev.yaml
        ├── values-qa.yaml
        └── values-prod.yaml

Each environment has its own values file, ensuring clean, controlled differences across Dev, QA, and Production.

Part 1 — Deploying the First Application

Accessing the Argo CD UI

The user logs into Argo CD, selects NEW APP, switches to the YAML editor, and pastes the leaderboard-dev Application manifest.

This manifest declares:

The repository location
Chart path
Values file (values-dev.yaml)
Destination namespace (dev)
Sync options such as automated namespace creation

First Sync

After clicking CREATE, the Application appears as Missing and OutOfSync. A manual SYNC triggers Kubernetes to create Deployments, ReplicaSets, Pods, Services, and Endpoints. Once the Pod becomes Ready, the entire tree turns green.

Part 2 — Detecting and Understanding Drift

Configuration drift begins the moment Git changes but the cluster does not. When the image tag changes in Git, Argo CD detects the mismatch and marks the Deployment as OutOfSync, represented with a yellow icon.

Using the APP DIFF feature, differences become crystal clear—especially when filtering with:

Compact diff
Inline diff

This narrow view highlights only what's changed, such as an updated image tag.

Part 3 — Handling Suspended Deployments

Sometimes manual interventions happen. For example, imagine someone pauses a Deployment rollout directly in the cluster. Argo CD immediately reflects this by showing the Application as Suspended.

To resume:

Open the Deployment resource
Click the three-dot menu
Select Resume

Argo CD continues the rollout using the updated image tag.

Verification is simple—check the Pod’s SUMMARY tab for the expected image:

quay.io/akuity/.../leaderboard:0.6.1

pod_image_change_after_Resume

Part 4 — Scaling with the App of Apps Pattern

Creating Applications manually through the UI is useful for beginners but doesn’t scale. In production, Applications must be declared in Git.

The App of Apps pattern solves this elegantly.

A parent Application points to the /apps directory; Argo CD then automatically creates and manages all child Applications.

Key Advantages

All apps are declared in Git
Adding new applications is as simple as committing a YAML file
Autosync ensures hands-free provisioning
Team onboarding becomes frictionless

Once auto-sync is enabled for the parent app, the entire environment becomes self-managing.

Part 5 — Production Troubleshooting: ImagePullBackOff

Production failures are inevitable. One common case:
An image tag is incorrectly specified. For example:

Argo CD shows Pods stuck in ImagePullBackOff.

A temporary fix may require editing the live manifest:

Open the Deployment
Switch to LIVE MANIFEST
Remove the v prefix

Save

  v0.6.1  ❌   (invalid because of the ‘v’ prefix)
  0.6.1   ✔️   (correct)

The cluster recovers immediately.
But Git still contains the wrong value, so Argo CD will continue to detect drift.

Permanent Fix

Update values-prod.yaml:

image:
  tag: "0.6.1"

production back up and running

Commit → push → Argo CD syncs → drift resolved.

Key Takeaways

1. 🧭 Git is the Source of Truth

All changes must flow through Git — not kubectl.

2. 🛠️ Helm Simplifies Multi-Environment Delivery

Values files keep overrides clean, declarative, and predictable.

3. 🔍 Argo CD Detects Drift Relentlessly

No discrepancy escapes its sync engine.

4. 🗂️ The App-of-Apps Pattern Scales Beautifully

Large fleets of applications become easy to manage.

5. ⚡ Hotfixes Are Temporary

Every manual patch must be followed by the corresponding Git update.

Best Practices Summary

Commit every change to Git—no exceptions
Use environment-specific Helm values
Enable auto-sync for stable environments
Follow up hotfixes with commits
Organize manifests cleanly
Keep Applications declarative

Conclusion

This guide demonstrates a complete, real-world GitOps workflow for managing multi-environment Kubernetes applications with Argo CD and Helm. From initial deployment to drift detection, from suspended rollouts to production fire-fighting, every step reinforces one truth:

GitOps isn’t just a methodology—it’s a discipline.

Kubernetes RBAC in Action: Built a Superhero Admin Group⚔️

Hardik Arora — Fri, 21 Nov 2025 18:16:10 GMT

A concise, hands-on guide to creating and verifying a custom super-user group in Kubernetes — complete with certificate-based identity and real-world access validation.

🧩 How Default Admin Works

Component	Value
User (CN)	`kubernetes-admin`
Group (O)	`system:masters`
Binding	`system:masters → cluster-admin`
Permissions	`` verbs on `` resources
Result	Full cluster access via certificate

🦸‍♂️ Create Superhero Admin Group

1️⃣ Create a Full-Access ClusterRole, and bind it to the group

2️⃣3️⃣ Test Permissions with `can-i`

kubectl auth can-i '*' '*' --as=batman --as-group=cluster-superheroes
# → yes

Same goes for superman, wonder-woman, or any member of cluster-superheroes.

🔐 Prove It Works — Real User with Signed Certificate

Step 1: Generate Key & CSR

openssl req -new -key batman.key -out batman.csr \
  -subj "/CN=batman/O=cluster-superheroes" -sha256
[Image: cert-sign-request.png]

Step 2: Submit & Approve CSR

Encode the CSR data and create a CSR manifest:

CSR_DATA=$(base64 batman.csr | tr -d '\n')

cat < batman-csr.yaml
apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  name: batman
spec:
  request: $CSR_DATA
  signerName: kubernetes.io/kube-apiserver-client
  usages:
  - client auth
EOF

kubectl apply -f batman-csr.yaml

Now approve the CSR as an admin:

Step 3: Extract Certificate and Verify

kubectl get csr batman -o jsonpath='{.status.certificate}' | base64 -d > batman.crt

openssl x509 -in batman.crt -text -noout | grep -E "Subject|Issuer"
# → CN=batman, O=cluster-superheroes

🧭 Add Batman to KubeConfig

kubectl config set-credentials batman \
  --client-certificate=batman.crt --client-key=batman.key --embed-certs=true

kubectl config set-context batman-context \
  --cluster=$(kubectl config get-clusters | grep -v NAME | head -1) \
  --user=batman

kubectl config use-context batman-context

🧠 Real Proof — Try Admin Actions

If these all succeed — congratulations, Batman user is added!

✅ Best Practices

Do	Avoid
Use groups, not individual users	Adding to `system:masters` in production
Rotate certificates yearly	Manual CSRs long-term
Audit role bindings regularly	Overly broad roles

📘 Summary

Step	Outcome
Certificate → Group	`O=cluster-superheroes`
Group → Binding	`cluster-superhero`
Role → Permissions	`` on ``

How Argo CD Actually Syncs Kubernetes Manifests — Under the Hood💫

Hardik Arora — Tue, 11 Nov 2025 12:57:01 GMT

Ever clicked “Sync” in Argo CD and wondered what’s really going on behind that magic button?
Turns out, it’s not just pushing YAMLs to your cluster — it’s running a smart control loop that keeps your live state perfectly aligned with your Git state.

Let’s break down how it actually works — piece by piece — from a DevOps learner’s perspective.

⚙️ 1. The Argo CD Application — Your GitOps Blueprint

Everything starts with an Application — basically Argo CD’s definition of what to deploy and where.

It has three main parts:

Source → where the manifests come from (Git repo, Helm chart, etc.)
Destination → which cluster and namespace to apply them to
Sync Policy → how updates should happen (manual or automatic)

Once that’s in place, Argo CD watches both your Git repo and your cluster — constantly checking if they’re in sync.

That’s GitOps 101: Git = truth, cluster = reality.

☸ 2. Manifest Generation — Turning Source into Deployable YAML

When an Application is created, Argo CD figures out how to turn your source into raw manifests:

Auto-Detection → figures out if you’re using Helm, Kustomize, or plain YAML.
Rendering →
- Helm → helm template
- Kustomize → kustomize build
- YAML → just reads it as is

What you get is a clean, ordered list of Kubernetes resources ready for deployment.

🧾 3. Normalization & Label Magic

Before applying anything, Argo CD cleans things up a bit:

Sorts resources in the right order (CRDs first, then CRs).
Adds internal labels like:
```
  argocd.argoproj.io/instance: my-app
```

These tiny annotations are how Argo CD tracks which resources belong to which app — and later detects drift or cleans up old ones.

🔍 4. The Real “Sync” — Comparing Desired vs. Live

Here’s where the magic happens.
Argo CD compares what’s in Git with what’s actually running in the cluster.

If something’s off, it highlights the difference.
In manual mode, you decide when to fix it.
In auto mode, Argo CD just goes ahead and reconciles.

During sync, it:
✅ Creates missing resources
🩹 Updates the changed ones
🧹 Deletes anything removed from Git

It’s basically kubectl apply --prune — but way smarter, safer, and Git-driven.

image courtesy: Akuity

❤️ 5. Health Checks — Because “Running” Isn’t Always “Healthy”

After syncing, Argo CD checks how your resources are doing:

Status	What It Means
Healthy	Everything’s good to go
Progressing	Still creating or updating
Degraded	Something’s broken (e.g. pod crash)
Missing	Expected resource isn’t there
Suspended	Waiting for an external event

Your app’s overall health is just the combined status of all these — a simple but super useful view when debugging.

🧠 6. Self-Healing — Argo CD as Your Cluster’s Immune System

Enable selfHeal in your syncPolicy and Argo CD instantly becomes your cluster’s watchdog.

Change something manually with kubectl? Argo CD will roll it back.
Delete a resource from Git? It’ll remove it from the cluster too.

It’s the heart of GitOps — no manual drift, no hidden state.
Your cluster always reflects what’s in Git, nothing else.

🌐 7. Multi-Cluster Power — One Argo, Many Worlds

Using the destination field, you can decide exactly where things go:

You can deploy:

To your local cluster, or
To any other connected cluster via argocd cluster add

This makes Argo CD awesome for managing dev → staging → prod pipelines or even multi-region setups.

📣 8. Notifications & Feedback Loops

Every sync, health update, or failure triggers events that can notify you via:

Slack / MS Teams
Webhooks
CI/CD pipelines like Tekton or Argo Workflows

It’s not just automation — it’s observability with context.

✨ TL;DR — The GitOps Control Loop

Argo CD is the heartbeat of GitOps.
It continuously watches, compares, and corrects — keeping your cluster honest, one sync at a time.

Final Thoughts

When I first hit “Sync,” I thought it was just another deploy.
Now I know it’s a whole control loop — a living system that aligns my cluster with my intent.

That’s what makes Argo CD so addictive.
It’s not just deploying YAMLs — it’s watching your ideas come alive, in real time, the GitOps way.

Big thanks to the folks at Akuity for their amazing Argo CD course — it helped me finally understand what really happens after hitting “Sync.” 🙌☁️

🧠 Talking to Kubernetes Without kubectl — My First Raw API Experiment

Hardik Arora — Fri, 07 Nov 2025 21:32:48 GMT

Today I created and deleted a Kubernetes Pod without using kubectl — just curl and the Kubernetes API itself.
And that small act completely changed how I see the cluster.

⚙️ Setting the Stage

It started with a question — what really happens when I run kubectl run nginx?
We all know kubectl talks to the API server, but I wanted to see it happen.

So, I decided to reverse-engineer the entire flow.

🔍 Reverse-Engineering kubectl

Here’s the chain of logic that led to the curl command:

kubectl is a wrapper around the Kubernetes API.
Every command you run — kubectl get pods, kubectl apply, kubectl run nginx — is just an HTTP call under the hood.
Reveal the real API call using verbosity (-v=6).
```
 kubectl run nginx --image=nginx -v=6
```
This displays the exact REST request kubectl sends — a POST to
```
 https:///api/v1/namespaces/default/pods
```
with a JSON body describing the Pod.
Recreate it manually with kubectl proxy.
```
 kubectl proxy &
```
This starts a local HTTP proxy at:
```
 http://localhost:8001
```
It tunnels traffic to the API server using your kubeconfig credentials.
Inspect and export the request using Postman.
Point Postman to:
```
 http://localhost:8001/api/v1/namespaces/default/pods
```
Set method to POST, add headers:
```
 Accept: application/json  
 Content-Type: application/json
```
and paste your Pod JSON body.
Then click Code → cURL — and voilà, you get the real REST call kubectl makes.

🚀 Creating a Pod Without kubectl

Here’s the final command that came out of this reverse-engineering:

curl -X POST http://localhost:8001/api/v1/namespaces/default/pods?pretty=true \
  -H "Accept: application/json" \
  -H "Content-Type: application/json" \
  -d '{
    "apiVersion": "v1",
    "kind": "Pod",
    "metadata": { "name": "nginx" },
    "spec": {
      "containers": [{ "name": "nginx", "image": "nginx:latest" }]
    }
  }'

The response:

HTTP/1.1 201 Created

A Pod was born — no YAML apply, no kubectl magic. Just pure REST.

🧹 Deleting the Pod

curl -X DELETE http://localhost:8001/api/v1/namespaces/default/pods/nginx

And Kubernetes instantly acknowledged it.
At that moment, I stopped seeing kubectl as a “tool” — it’s simply an API client.

🧩 What Happens Behind the Scenes

Every API request to Kubernetes travels through these key steps:

Request Arrival → API server receives it (default port 6443).
Routing → /api/v1/namespaces/default/pods maps to the Pod handler.
Authentication → Verifies who is making the request.
Authorization (RBAC/Webhooks) → Checks what that identity can do.
Admission Control → The heart of the control plane:
- Istio injects the Envoy sidecar via its mutating webhook.
- OPA Gatekeeper enforces policies like disallowing :latest images.
Validation → Ensures schema correctness.
Persistence → Writes the object to etcd and triggers controllers.
Response → Returns JSON with a status like 201 Created or 200 OK.

That’s the full roundtrip your tiny curl just made across the Kubernetes control plane.

Every operator, controller, or GitOps tool — from Argo CD to Istio — is just another API client like your curl.
Once you understand that, cluster internals start making real sense.

Deploying and Scaling a Kubernetes Guestbook Application

Hardik Arora — Tue, 19 Aug 2025 16:52:41 GMT

This project demonstrates how to build, deploy, and scale a simple Guestbook application on Kubernetes. The workflow covers containerization, autoscaling, and safe rollout practices—core competencies for cloud-native operations.

Step 1: Build and Deploy the Guestbook Image

We begin by building and pushing the application image:

# install dependencies
FROM golang:1.18 AS builder
#set working directory
WORKDIR /app
# copy script 
COPY main.go .
# commands to initialise project
RUN go mod init guestbook
RUN go mod tidy
RUN go build -o main main.go
# container image
FROM ubuntu:18.04 
#Copy all files to image
COPY --from=builder /app/main /app/guestbook
COPY public/index.html /app/public/index.html
COPY public/script.js /app/public/script.js
COPY public/style.css /app/public/style.css
COPY public/jquery.min.js /app/public/jquery.min.js
WORKDIR /app
CMD ["./guestbook"]
# expose the container port to listen to requests
EXPOSE 3000

deployment.yaml stating all deployment configurations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: guestbook
spec:
  replicas: 1
  selector:
    matchLabels:
      app: guestbook
  strategy:
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: guestbook
    spec:
      containers:
      - name: guestbook
        image: us.icr.io/sn-labs-arorahardik0/guestbook:v1
        ports:
        - containerPort: 3000

Docker commands to build image and push to IBM cloud registry:

docker build -t us.icr.io/$MY_NAMESPACE/guestbook:v1 .
docker push us.icr.io/$MY_NAMESPACE/guestbook:v1

Best Practice Validation

Following a method recommended by Google employees, I validated that the image in the manifest existed and could be pulled:

kubectl get events -n $MY_NAMESPACE --sort-by=.metadata.creationTimestamp

kubectl apply -f deployment.yaml

This prevents invalid or broken deployments caused by incorrect image references.

Step 2: Autoscaling with HPA

Next, I enabled the Horizontal Pod Autoscaler (HPA) to automatically adjust replicas based on load:

kubectl autoscale deployment guestbook --cpu-percent=50 --min=1 --max=5

To simulate load and test scaling behavior:

kubectl run -i --tty load-generator --rm \
  --image=busybox:1.36.0 --restart=Never -- \
  /bin/sh -c "while sleep 0.01; do wget -q -O- https://arorahardik0-3000.theiaopenshiftnext-1-labs-prod-theiaopenshift-4-tor01.proxy.cognitiveclass.ai/; 
done"

I then observed scaling in real time:

Step 3: Rolling Updates and Rollbacks

I tested deployment resilience by updating the app (v2 image). After pushing the updated image, I changed the deployment.yaml and re-applied:

kubectl apply -f deployment.yaml

Rollout History and Rollback

View rollout history:

kubectl rollout history deployment/guestbook

Then getting details for a particular version:

If required, rollback safely to a previous version:

kubectl rollout undo deployment/guestbook --to-revision=1

This guarantees production safety by allowing quick recovery from failed updates.

Key Takeaway

Autoscaling: HPA ensures applications adapt to varying workloads without manual intervention.
Operational Safety: Rolling updates and rollbacks safeguard against failures.
Validation: Using kubectl get events confirms image pullability before rollout.

This project shows how even a simple app like Guestbook can highlight enterprise-grade Kubernetes workflows: build → deploy → scale → update.