Routine operations
Use this guide for a read-only health pass or to find where a failure starts. It ends where a change would begin: every live action needs approval and its own procedure.
Before you start
Confirm which cluster you are talking to, and use the repository's tool versions:
mise exec -- kubectl config current-context
mise exec -- task --list
Run commands through mise exec so they use the pinned versions. Take task names from task --list, not from old notes.
Read-only health pass
Check from the bottom up: nodes, then Flux, then the workload you care about.
mise exec -- kubectl get nodes
mise exec -- flux get sources all -A
mise exec -- flux get kustomizations -A
mise exec -- flux get helmreleases -A
mise exec -- kubectl -n <namespace> get pods,events
For the object that looks wrong, run kubectl describe and a time-bounded kubectl logs --since=<window>. Write down the condition reason and message, the generation, the restart count, and the event times before anything changes them. Redact anything secret before sharing output.
Find the first failing layer
A broken pod is often a symptom. Walk down the chain and stop at the first layer that is not healthy:
Then use the guide for that layer:
| Symptom | Guide |
|---|---|
| Node not Ready, disk, interface, or etcd trouble | Talos and node health |
| Kustomization or HelmRelease not Ready | Flux remediation |
| Volume attach or storage network failure | Storage |
| Missing or delayed alerts | Alert delivery |
| Missing logs | Kubernetes logs |
Rebuild readiness
scripts/check-rebuild-prerequisites.py audits, offline, whether committed sources describe a dependency-ordered rebuild. It never contacts the cluster or reads secrets. PASS means a declared source exists, and UNKNOWN means Git alone cannot prove availability. The current audit flags a possible cold-start cycle: a storage-network image is served from Harbor, which starts later. A clean audit is not a restore test. The rebuildability reference includes a two-hour tabletop exercise.
Changing anything
Force reconciliation, restarts, port-forwards, debug pods, storage changes, and Talos applies are live actions. Each needs explicit operator approval, the matching subsystem procedure, and a stated expected result. A passing repository validation is not, by itself, a reason to apply.