ArgoCD was already Helm-owned. Single Redis was healthy, but it was the last obvious single point of failure in the control plane. Enabling redis-ha.enabled was supposed to be a values change.

It was not.

The symptom

The first rollout reached two Redis pods and two HAProxy pods. The third Redis pod stayed in init:

argocd-redis-ha-server-0   0/3   Init:0/1
argocd-redis-ha-haproxy    2/3   Running

The init log was not a Redis crash:

Identifying redis master (get-master-addr-by-name)..
  using sentinel (argocd-redis-ha), sentinel group name (argocd)
Could not connect to Redis at argocd-redis-ha:26379: Try again

HAProxy was waiting for the same sentinel. The announce endpoint existed, but connecting to it timed out. The pod could not become ready, so the sentinel Service had no usable endpoint, so the lookup kept failing.

First, rule out the test cluster

I built three disposable NixOS VMs on calamansi-intel to test this without touching production. The first test was invalid.

k3s picked the wrong network interface for Flannel:

flannel.1 local 10.0.2.15 dev eth0

That is the QEMU NAT address, not the cluster network. CoreDNS was unreachable across nodes, and Redis init hung on getent hosts. The fix was:

--flannel-iface=eth1

The k3s Flannel interface option is documented here: https://docs.k3s.io/networking/basic-network-options#flannel-options

After reboot, DNS and cross-node pod traffic worked. The Redis failure remained. That ruled out CNI, scheduling and resources.

The real problem

The chart creates one announce Service per Redis pod, but it hardcodes them as ClusterIP Services:

spec:
  publishNotReadyAddresses: true
  type: ClusterIP

config-init resolves the announce name and gets the Service VIP, not the pod IP. Sentinel then monitors that VIP. The split-brain container never converges, so the pod stays NotReady.

This is upstream issue:

https://github.com/DandyDeveloper/charts/issues/422

The related masterless deadlock report is: https://github.com/DandyDeveloper/charts/issues/412

The fix proposed there is simple: make the per-pod announce Services headless. Kubernetes documents headless Services here: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services

spec:
  clusterIP: None
  publishNotReadyAddresses: true

I verified it on the test cluster. Announce resolved to a pod IP, Redis reached 3/3, HAProxy reached 3/3, and deleting the master made sentinel promote another pod. The old master rejoined as a replica.

Making it repeatable

A manual kubectl patch would be undone by the next Helm upgrade. The chart has no value for this, so I added a Helm post-renderer. It rewrites only the redis-ha announce Services.

The script reads Helm output on stdin and inserts clusterIP: None after spec for any resource whose name matches -announce-[0-9]+:

awk '
  /^---[[:space:]]*$/ { announce=0; print; next }
  /^  name:[[:space:]]+.*-announce-[0-9]+[[:space:]]*$/ { announce=1 }
  /^spec:[[:space:]]*$/ && announce { print; print "  clusterIP: None"; next }
  { print }
'

The values also need a toleration for the tainted control-plane node. The chart reads redis-ha.tolerations at the top level, not under redis or haproxy. That detail cost one more rollout.

redis-ha:
  enabled: true
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule

The deploy

Helm 4 no longer accepts an executable post-renderer, so this used Helm 3.17.3:

helm -n argocd upgrade --install argocd argo/argo-cd \
  --version 10.10.5 -f manifests/argocd/values.yaml \
  --post-renderer ./scripts/argocd-redis-ha-post-renderer.sh \
  --take-ownership --wait --timeout 12m

The Redis server ConfigMap changed, but the existing ArgoCD pods kept their old cache endpoint. Restarting the consumers fixed the last connection errors:

kubectl -n argocd rollout restart \
  deploy/argocd-repo-server deploy/argocd-server \
  deploy/argocd-applicationset-controller \
  statefulset/argocd-application-controller

Verify

kubectl -n argocd get pods -l 'app.kubernetes.io/name in (argocd-redis-ha,argocd-redis-ha-haproxy)'
kubectl -n argocd get endpoints argocd-redis-ha argocd-redis-ha-haproxy
kubectl -n argocd get app root

Production result:

• Redis 3/3 across all three nodes;
• HAProxy 3/3 across all three nodes;
• both Services have three endpoints;
• ArgoCD root is Synced and Healthy;
• 29 applications Synced and Healthy, with the existing Cilium drift still showing as OutOfSync but Healthy.

The chart still owns the workaround, not the fix. The post-renderer has to be included on every upgrade, and the consumers need a restart after the Redis endpoint changes. The rollback is still a Helm rollback to the single Redis revision.

Links

• Upstream headless announce issue: https://github.com/DandyDeveloper/charts/issues/422
• Previous post, official HA manifest route: ArgoCD Redis HA on a tainted k3s cluster
• Helm post-renderers: https://helm.sh/docs/topics/advanced/#post-rendering
• ArgoCD high availability: https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/
• Production GitOps changes: bakuran-gitops PR #124, #125 and #126 (the repository is private, so there is no public link).

Update 2026-10-20: This post documents the official-manifest attempt. Production later moved to the Helm chart with a headless announce post-renderer. The follow-up is Getting ArgoCD Redis HA through the chart, not around it .

ArgoCD stores desired state in etcd. Redis is a cache. Losing Redis pauses reconciliation, but it does not take down running workloads. That made the single Redis pod easy to ignore, until it became the last obvious single point of failure in the control plane.

This is the retry that worked.

The setup

The cluster has three k3s nodes. Two are general schedulers. The third, kamatis-mbp, is control-plane and etcd only, with a NoSchedule taint.

I used the official ArgoCD HA Redis resources, not the previous third-party chart attempt:

• three Redis servers with sentinel;
• three HAProxy pods;
• headless announce services for sentinel discovery;
• ArgoCD pointed at argocd-redis-ha-haproxy:6379.

The official HA manifest is here: https://github.com/argoproj/argo-cd/blob/master/manifests/ha/install.yaml

What failed

Redis was healthy on two nodes. The third pod stayed Pending:

0/3 nodes are available:
  1 node(s) had untolerated taint(s),
  2 node(s) didn't match pod anti-affinity rules.

The HA manifests use required pod anti-affinity so Redis and sentinel are spread across three nodes. That is correct for quorum. But the third node was tainted, so the scheduler had no legal place to put the third replica.

The first HAProxy pod hit the same wall. During the rollout, old HAProxy pods also held their nodes, so a replacement stayed Pending even after the new ReplicaSet was ready.

The fix

Add a toleration for the control-plane taint to both the Redis StatefulSet and the HAProxy Deployment:

TOL='[{"key":"node-role.kubernetes.io/control-plane","operator":"Exists","effect":"NoSchedule"}]'

kubectl -n argocd patch statefulset argocd-redis-ha-server --type merge \
  -p "{\"spec\":{\"template\":{\"spec\":{\"tolerations\":$TOL}}}}"

kubectl -n argocd patch deploy argocd-redis-ha-haproxy --type merge \
  -p "{\"spec\":{\"template\":{\"spec\":{\"tolerations\":$TOL}}}}"

Then remove the stale rolling pods so the scheduler can place the replacements. Finally point ArgoCD at HAProxy:

kubectl -n argocd patch configmap argocd-cmd-params-cm --type merge \
  -p '{"data":{"redis.server":"argocd-redis-ha-haproxy:6379"}}'

kubectl -n argocd rollout restart \
  deploy/argocd-server deploy/argocd-repo-server \
  deploy/argocd-applicationset-controller \
  statefulset/argocd-application-controller

Verify

kubectl -n argocd get pods -l 'app.kubernetes.io/name in (argocd-redis-ha,argocd-redis-ha-haproxy)'
kubectl -n argocd get app root
kubectl -n argocd logs deploy/argocd-repo-server --tail=50

Expected:

• three Redis server pods at 3/3;
• three HAProxy pods at 1/1;
• ArgoCD root app Synced and Healthy;
• no Redis connection errors in the repo server.

At the time, the old single Redis deployment was kept as a fallback. The follow-up replaced this path with chart-managed HA; the fallback is now a Helm rollback revision, not a running second Redis.

Why this was not a Redis bug

The failure was scheduling policy, not Redis, sentinel or ArgoCD. The official HA manifest assumes three schedulable nodes. A tainted control-plane node needs an explicit toleration. That is a cluster decision, not a chart defect.

The bootstrap script now applies the HA resources, adds the toleration and sets the HAProxy endpoint, so a rebuild does not depend on manual patches.

Sentinel CPU spin

Redis HA has a known sentinel bug. After a restart, one or two sentinel containers can pin a full CPU core with no useful log output. ArgoCD stays healthy, but the core burn is real.

The workaround is a delayed sentinel reset after startup:

lifecycle:
  postStart:
    exec:
      command: ["/bin/sh", "-c", "sleep 30; redis-cli -p 26379 sentinel reset argocd"]

The official-manifest bootstrap applied that to the sentinel container. The Helm chart path does not. There, container CPU alerts and the Redis HA alerts are the safety net; apply the same reset by hand if a sentinel pins a core. It is a workaround, not an upstream fix.

• Redis bug: https://github.com/redis/redis/issues/9956
• ArgoCD bug: https://github.com/argoproj/argo-cd/issues/16360
• Patch reference: https://github.com/utilitywarehouse/system-manifests/pull/286

Links

• ArgoCD high availability: https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/
• Official HA install manifest: https://github.com/argoproj/argo-cd/blob/master/manifests/ha/install.yaml
• Kubernetes taints and tolerations: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/
• The edge split and Cilium migration: Homelab network migration