The Office of the Night Watch

Nothing, until I have looked.

Establish first whether users are affected or a threshold was crossed. These are not the same event and only one of them is worth your night.

A disk at 91% is not an outage. A disk at 91% filling by one point a minute is an outage that has not happened yet, and the difference between those two costs sixty seconds:

df -h; sleep 60; df -h

Ask the layer below before you accuse your own.

Three commands, in this order, before touching anything you wrote:

kubectl get nodes
kubectl get events -A --sort-by=.lastTimestamp | tail -30
kubectl get pods -A --field-selector=status.phase!=Running

If several unrelated services broke at the same second, the thing they share is not the code. It is the node, the network, the registry or the clock.

Something did. It always did.

Almost nothing breaks spontaneously. Find the change before you find the fault, and search wider than your own deploys: a certificate expiring is a change, a cron job that ran for the first time this month is a change, and so is a dependency that upgraded itself because somebody left it on a floating tag.

kubectl rollout history deployment/<name>
kubectl get events -A --sort-by=.lastTimestamp | grep -i -E 'scaled|updated|created'

Mitigate. Understanding keeps until morning. Users do not.

This is the hour operators get wrong in both directions. The rule is not complicated: if you can restore service without destroying the evidence, do it now. A rollback is almost always that. A restart sometimes is not, because it takes the state you needed with it.

Before you restart anything, take the thing you cannot get back afterwards:

kubectl logs <pod> --previous > /tmp/incident.log
kubectl describe pod <pod> >> /tmp/incident.log

The one who owns it, not the one who answers.

Escalating is not an admission. It is the correct action at a known threshold, and the threshold should be a clock, not a feeling: if you have not materially advanced in twenty minutes, wake somebody. The instinct to keep trying alone is the most expensive instinct in operations.

Say four things and nothing else. What is broken. Since when. What you have already tried. What you need from them.

Watch it for longer than feels necessary.

The failure mode of this hour is declaring victory on one green check. Systems that recover on their own also fail on their own, and a backoff that has reached five minutes will look healthy for four of them.

watch -n 30 'kubectl get pods -n <ns>'

Stay until you have seen it survive one full cycle of whatever was killing it. If you do not know what that cycle is, you have not finished.

Not what I did. What is still true.

Write it while it is dull rather than while it is dramatic. Three lines is enough and three lines beats a perfect document written never:

• What is currently mitigated, and how, and whether the mitigation is temporary.
• What you ruled out, so nobody spends an hour on it again.
• What you did not check.

The third line is the one people leave out and the only one that is hard to reconstruct in the morning.

If it is mitigated, written down, and owned by somebody awake, yes.

All three, or none. A mitigation nobody knows about is not a mitigation, it is a trap you have set for the morning shift.

And then actually sleep. The last hour of a bad night produces the changes that cause the next one. There is no version of this job where being awake for nineteen hours makes the cluster better, and every incident review that finds a tired person making a rushed change finds it at exactly this hour.