bismigel

Getting ArgoCD Redis HA through the chart, not around it

· 4 min read

The previous post used the official ArgoCD HA manifests and solved a scheduling problem. This one is about the Helm chart. The chart has a different failure, and it took a disposable cluster to see it clearly.

ArgoCD was already Helm-owned. Single Redis was healthy, but it was the last obvious single point of failure in the control plane. Enabling redis-ha.enabled was supposed to be a values change.

It was not.

The symptom

The first rollout reached two Redis pods and two HAProxy pods. The third Redis pod stayed in init:

text 
argocd-redis-ha-server-0   0/3   Init:0/1
argocd-redis-ha-haproxy    2/3   Running

The init log was not a Redis crash:

text 
Identifying redis master (get-master-addr-by-name)..
  using sentinel (argocd-redis-ha), sentinel group name (argocd)
Could not connect to Redis at argocd-redis-ha:26379: Try again

HAProxy was waiting for the same sentinel. The announce endpoint existed, but connecting to it timed out. The pod could not become ready, so the sentinel Service had no usable endpoint, so the lookup kept failing.

First, rule out the test cluster

I built three disposable NixOS VMs on calamansi-intel to test this without touching production. The first test was invalid.

k3s picked the wrong network interface for Flannel:

text 
flannel.1 local 10.0.2.15 dev eth0

That is the QEMU NAT address, not the cluster network. CoreDNS was unreachable across nodes, and Redis init hung on getent hosts. The fix was:

text 
--flannel-iface=eth1

The k3s Flannel interface option is documented here: https://docs.k3s.io/networking/basic-network-options#flannel-options

After reboot, DNS and cross-node pod traffic worked. The Redis failure remained. That ruled out CNI, scheduling and resources.

The real problem

The chart creates one announce Service per Redis pod, but it hardcodes them as ClusterIP Services:

yaml 
spec:
  publishNotReadyAddresses: true
  type: ClusterIP

config-init resolves the announce name and gets the Service VIP, not the pod IP. Sentinel then monitors that VIP. The split-brain container never converges, so the pod stays NotReady.

This is upstream issue:

https://github.com/DandyDeveloper/charts/issues/422

The related masterless deadlock report is: https://github.com/DandyDeveloper/charts/issues/412

The fix proposed there is simple: make the per-pod announce Services headless. Kubernetes documents headless Services here: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services

yaml 
spec:
  clusterIP: None
  publishNotReadyAddresses: true

I verified it on the test cluster. Announce resolved to a pod IP, Redis reached 3/3, HAProxy reached 3/3, and deleting the master made sentinel promote another pod. The old master rejoined as a replica.

Making it repeatable

A manual kubectl patch would be undone by the next Helm upgrade. The chart has no value for this, so I added a Helm post-renderer. It rewrites only the redis-ha announce Services.

The script reads Helm output on stdin and inserts clusterIP: None after spec for any resource whose name matches -announce-[0-9]+:

bash 
awk '
  /^---[[:space:]]*$/ { announce=0; print; next }
  /^  name:[[:space:]]+.*-announce-[0-9]+[[:space:]]*$/ { announce=1 }
  /^spec:[[:space:]]*$/ && announce { print; print "  clusterIP: None"; next }
  { print }
'

The values also need a toleration for the tainted control-plane node. The chart reads redis-ha.tolerations at the top level, not under redis or haproxy. That detail cost one more rollout.

yaml 
redis-ha:
  enabled: true
  tolerations:
    - key: node-role.kubernetes.io/control-plane
      operator: Exists
      effect: NoSchedule

The deploy

Helm 4 no longer accepts an executable post-renderer, so this used Helm 3.17.3:

bash 
helm -n argocd upgrade --install argocd argo/argo-cd \
  --version 10.10.5 -f manifests/argocd/values.yaml \
  --post-renderer ./scripts/argocd-redis-ha-post-renderer.sh \
  --take-ownership --wait --timeout 12m

The Redis server ConfigMap changed, but the existing ArgoCD pods kept their old cache endpoint. Restarting the consumers fixed the last connection errors:

bash 
kubectl -n argocd rollout restart \
  deploy/argocd-repo-server deploy/argocd-server \
  deploy/argocd-applicationset-controller \
  statefulset/argocd-application-controller

Verify

bash 
kubectl -n argocd get pods -l 'app.kubernetes.io/name in (argocd-redis-ha,argocd-redis-ha-haproxy)'
kubectl -n argocd get endpoints argocd-redis-ha argocd-redis-ha-haproxy
kubectl -n argocd get app root

Production result:

  • Redis 3/3 across all three nodes;
  • HAProxy 3/3 across all three nodes;
  • both Services have three endpoints;
  • ArgoCD root is Synced and Healthy;
  • 29 applications Synced and Healthy, with the existing Cilium drift still showing as OutOfSync but Healthy.

The chart still owns the workaround, not the fix. The post-renderer has to be included on every upgrade, and the consumers need a restart after the Redis endpoint changes. The rollback is still a Helm rollback to the single Redis revision.

Links

homelab kubernetes argocd redis