Building the lab: side story. (Not part of the numbered series.) ← Homelab observability: Prometheus, Grafana and five focused dashboards · Next: Homelab epilogue: exposing services and publishing the repo

The lab has an SRE now. That sentence sounds more dramatic than the reality. Hermes is an AI agent that runs inside the cluster, reads what is happening, and writes a short report. It does not replace the pager, and it does not merge its own work.

I did not set out to build an AI SRE. I wanted fewer surprises. The lab already had Prometheus, Grafana, Alertmanager and a pile of runbooks. What it did not have was something that would look across all of them, notice a small problem, and tell me in one place. That is the gap Hermes fills.

Why an agent instead of another dashboard

I have five dashboards. Each one answers a different question. That is good when I am debugging, and bad when I just want to know whether the lab is healthy.

A dashboard waits for me to look. An alert tells me when a line is crossed. Neither one explains what it found. I wanted a first responder that could check the nodes, query Prometheus, read a pod log, look at the ArgoCD state, and put the result into one short message.

That is an SRE in the small sense. It watches, triages and reports. I still make the changes.

What Hermes does

• Every morning at 08:00 America/Toronto, it checks nodes, pods, restarts, ArgoCD sync and health, firing alerts, filesystem use and memory trends. It sends a short report to Telegram.
• When I ask, it can look at a deployment, read pod logs, query Prometheus, check ArgoCD, inspect the media queue, or search the book stack.
• It can open a pull request for a fix. The rule is branches and pull requests, never main.
• It can add a series or a book to the media queue. The media tools have no delete action, so it cannot remove anything.
• It can read Cloudflare state: tunnels, zones, DNS, certificates and Access apps. It cannot change any of it.

The daily report is the part that changed how I use the lab. It is not a wall of graphs. It is a short list of what is healthy and what is not, and it lands where I already look.

The architecture

Hermes runs as a Deployment in the cluster. It talks to the lab through MCP servers, and each server is a boundary around a set of actions.

The MCP layer matters. MCP (Model Context Protocol) is a standard way to give the agent a tool with its own permissions. I can give it a useful action without giving it a shell command for everything. The Kubernetes server uses a read-only ServiceAccount. The media server has no delete action. The Cloudflare path is an nginx proxy that only allows GET, HEAD and OPTIONS.

The agent still has a shell for kubectl and gh. That is the part that makes it useful, and the part that makes the guardrails matter.

How to set it up

The setup is boring on purpose. None of it is clever, and the boring parts are the ones that keep it safe.

1. Run it as a workload. Deploy Hermes as a normal Deployment with a PVC for state at /opt/data. The state includes config, sessions, memory and the cron list.
2. Give it a face. Connect a Telegram bot with a single-user allowlist, and put the dashboard behind Cloudflare Access and basic auth.
3. Give it context. Mount the operating docs and a SOUL.md rules file. An agent without context is a confident guesser.
4. Add MCP servers. Kubernetes, Prometheus, media and GitHub. Each server gets its own credentials and its own limits.
5. Use a dedicated ServiceAccount. Bind a read-only ClusterRole with get, list and watch. Do not give it Secrets.
6. Wrap Cloudflare. Put the token behind a sidecar that rejects every state-changing request with 403. The write-capable token stays on the host, outside the pod.
7. Scope GitHub. Use a repo-scoped token and make branches and pull requests the working rule. Do not give it permission to merge.
8. Seed a daily job. The 08:00 report is a prompt, not a fixed script, so it can adapt to what it finds. Ask for a short answer and no secrets.
9. Back up the state. The agent’s memory and job list are part of the appdata backup. A rebuilt agent should remember how the lab works.

The model is DeepSeek deepseek-chat through a custom OpenAI-compatible provider. The model is not the interesting part. The tools and the boundaries are.

The guardrails

The rules are in layers, and each layer assumes the one above it may fail.

The SOUL.md says the same rules in plain language: pull requests only, ask before anything destructive, never write secrets to a file or a log. The prompt is the last layer, not the first.

The hard limits are on the cluster and the edge. The Kubernetes role cannot change anything. The Cloudflare proxy cannot send a write. Those are technical blocks, not suggestions.

The GitHub rule is the soft spot. A repo-scoped token can still push to main if the agent decides to. That is why every branch and pull request gets a human look. I would rather say that plainly than pretend the prompt is a wall.

What Hermes does not do

• It does not apply changes to the cluster. It can read, explain and open a pull request.
• It does not merge its own pull requests.
• It does not write to Cloudflare.
• It does not delete media or books. The media tools have no delete action.
• It does not read Kubernetes Secrets.
• It does not replace Alertmanager. Alerts still go to me first.
• It does not run outside the lab. If the cluster is down, the agent is down too.

That last one is why the dead-man’s switch lives on the workstation outside the cluster. The agent is a second pair of eyes, not the only one.

Is an AI SRE worth it?

For this lab, yes, with conditions.

It saves me the morning check. It catches drift I would otherwise find days later. It is good at the boring part: look at five sources, compare them, and write one paragraph. It is also good at answering “what changed?” when I am already in the middle of a problem.

It is not free. It uses tokens, it can be wrong, and it needs context. A model without the docs will invent a topology that looks plausible and is not yours. The docs, the MCP tools and the guardrails are what make it useful.

The honest summary is this: I treat Hermes as a privileged operator with a narrow set of tools. It is allowed to look, explain, and propose. It is not allowed to finish a change on its own. That balance is the whole design.

If you want the lab details, the technical reference has the full setup. The epilogue explains why I am still nervous about exposing any of it.

Building the lab: side story. Previous: Homelab observability: Prometheus, Grafana and five focused dashboards . Next: Homelab epilogue: exposing services and publishing the repo .