The Twenty
I
Imago Non Inventa
ImagePullBackOff
The image is named, and the name means nothing, and the naming was in vain.
What it isThe kubelet asked a registry for your image and did not get it. Four causes, in the order they actually happen: the tag does not exist, the registry needs credentials you did not give it, the name has a typo, the registry is private and the node cannot reach it.
Look here firstkubectl describe pod <pod> and read the last event verbatim. It distinguishes manifest unknown from unauthorized, and those two have nothing to do with each other.
What wastes ten minutes
Rebuilding and repushing the image. If the tag is wrong, or the pull secret is missing, you can push all afternoon and the Pod will never see it.
II
Recursus Interminabilis
CrashLoopBackOff
The Pod dies, and rises, and dies again, and the interval between its deaths grows longer.
What it isThe container started and exited. Kubernetes restarted it, it exited again, and the backoff is doubling: ten seconds, twenty, forty, up to five minutes. The Pod is not broken. The Pod is obedient. It is doing exactly what you asked, repeatedly, and the thing you asked for does not work.
Look here firstkubectl logs <pod> --previous. The container running now has not said anything yet. The one that died has.
What wastes ten minutes
Reading the events. They will tell you it is in CrashLoopBackOff, which you knew before you typed anything. The answer is in the dead container's stdout, almost every time.
III
Memoria Consumpta
OOMKilled
It asked for more than was promised, and the kernel does not negotiate.
What it isThe container crossed its memory limit and the kernel killed it. Not Kubernetes. The kernel. This is why there is no graceful shutdown and no last log line: SIGKILL is not delivered to the process, it is delivered about the process.
Look here firstkubectl get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'. Exit code 137 confirms it.
What wastes ten minutes
Raising the limit as the first move. Sometimes right, often a leak you have just given more room to. Check whether the memory curve plateaus or climbs, and only then decide.
IV
Expectatio
Pending
It waits. Nothing is wrong. Nothing is happening.
What it isThe Pod exists in the API and no kubelet has been told to run it. Either the scheduler has not placed it, or it has been placed and the image is still coming down. These look identical from the outside and are not the same problem.
Look here firstkubectl describe pod <pod>. If Node: is empty it is the scheduler, see Sine Loco. If a node is named, it is the kubelet, and you are probably waiting on a pull.
What wastes ten minutes
Deleting and recreating it. If the cluster had nowhere to put it a minute ago, it has nowhere to put it now, and you have lost the events that would have said so.
V
Prohibitum
403 Forbidden
The gate is known, and the key is known, and the two do not know each other.
What it isSomething authenticated you and then refused you, which is the useful half of the message: a 403 means you were recognised. Inside the cluster this is RBAC, and the API server's version of it names the verb, the resource, the namespace and the identity in one sentence. From outside it is usually the ingress or the application, not Kubernetes at all.
Look here firstkubectl auth can-i <verb> <resource> --as=system:serviceaccount:<ns>:<sa>. It answers yes or no in one line, as the identity that is actually being refused rather than as you.
What wastes ten minutes
Granting cluster-admin to make it go away. It does make it go away. It also makes the next four months of least-privilege work impossible to justify.
VI
Non Inventum
404 Not Found
You have asked after something that was never here, or is here under another name.
What it isNothing was found at that name or that path. From kubectl it is nearly always the wrong namespace, because your context still points at whatever you were doing yesterday. From a browser it is nearly always the Ingress: the host matched, the path did not.
Look here firstFor kubectl, kubectl config current-context then kubectl get <kind> -A | grep <name>. For a 404 over HTTP, kubectl describe ingress <name> and read the rules as a list in order.
What wastes ten minutes
Reapplying the manifest. If you were in the wrong namespace you will now have two of the thing, in two places, and only one of them is wired up.
VII
Culpa Ignota
500 Internal Server Error
It has failed, and it will not say of what.
What it isThe application answered and the answer was that it broke. This is worth separating from its neighbours, because the number tells you who is speaking: 500 is the app, 502 and 504 are the proxy failing to reach it or waiting too long, and 503 is usually no ready endpoints at all. Only one of those four is a code problem.
Look here firstThe Pod's own logs, at the timestamp of the request. If the logs are silent, the 500 came from something in front of the Pod and you are debugging the wrong component.
What wastes ten minutes
Reading the ingress controller logs first. They will faithfully report that the backend returned 500, which you already knew, and will not tell you why.
VIII
Porta Clausa
Connection Refused
The door is where you were told, and it is shut, and nobody is behind it.
What it isSomething answered at that address and refused the port. That means DNS resolved and routing worked, which eliminates most of what people check. Either the process is not listening yet, or it is listening on 127.0.0.1 instead of 0.0.0.0, or the Service targets the wrong port.
Look here firstkubectl get endpoints <service>. Empty endpoints means the selector matches no ready Pod, and the Service is innocent.
What wastes ten minutes
Suspecting DNS. Connection refused is proof DNS worked. No such host is the DNS error, and it is a different one.
IX
Tempus Excessit
Timeout
The answer may yet come. You will not be here to receive it.
What it isSeen as context deadline exceeded in most Go components. A client gave up waiting. It says nothing about whether the server is broken, only that it was slower than somebody's timeout. Treat it as a question about which two components, and which of them owns the deadline.
Look here firstFind the deadline before you find the slowness. It is a probe timeout, a client config, or an ingress annotation, and until you know which one you are tuning blind.
What wastes ten minutes
Increasing every timeout you can find. That converts a fast clear failure into a slow unclear one, which is worse to be paged for.
X
Nunquam Extrahe
ErrImageNeverPull
You forbade the fetching, and then you asked for the thing you had not fetched.
What it isimagePullPolicy: Never and the image is not already on that node. Almost always a local development cluster where the image was built somewhere the node cannot see.
Look here firstLoad it into the node's own store: kind load docker-image, minikube image load, or the equivalent for your runtime.
What wastes ten minutes
Changing the policy to Always on a cluster with no registry. Now it will try to pull, and fail differently.
XI
Expulsus
Evicted
The node was in need, and it chose, and it did not choose you.
What it isThe kubelet reclaimed resources under pressure and your Pod was the one it picked. It picks by QoS class: BestEffort first, then Burstable over its request, and Guaranteed last. The Pod that gets evicted is usually the one that never declared what it needed.
Look here firstkubectl describe node <node> and read the conditions. DiskPressure and MemoryPressure name the resource that ran out.
What wastes ten minutes
Restarting the workload without setting requests. It will be first in the queue again next time, for the same reason.
XII
Terminatio Sine Fine
Terminating (stuck)
It was asked to go. It agreed. It remains.
What it isTwo different things wear this face. Either the container is ignoring SIGTERM and is inside its grace period, or the object has a finalizer and the controller that owns that finalizer is gone. The first resolves itself. The second never will.
Look here firstkubectl get <kind> <name> -o jsonpath='{.metadata.finalizers}'. Anything there and you are in the second case.
What wastes ten minutes
Reaching for --force --grace-period=0. It removes the object from the API and leaves whatever the finalizer was protecting still allocated, somewhere you are no longer looking.
XIII
Nodus Non Paratus
NodeNotReady
The place itself is unwell, and everything upon it is blamed.
What it isThe kubelet has stopped reporting, or reports that it is unhealthy. Every Pod on that node is about to look broken, and none of them is. Diagnosing upward from the Pod here can waste an hour.
Look here firstkubectl get nodes before anything else, every time. It is one command and it reframes the whole incident.
What wastes ten minutes
Debugging the application. If several unrelated services broke at once, the thing they share is not the code.
XIV
Processus Deest
Insufficient CPU
There is work in the house, and no hand free to do it.
What it isNo single node has enough unreserved CPU for what the Pod requested. Note requested, not used: the scheduler does arithmetic against requests and has no opinion at all about what is actually running. A cluster idling at 5% CPU can be completely unschedulable.
Look here firstkubectl describe node | grep -A 5 'Allocated resources'. The percentages there are of what is promised, which is the number the scheduler uses.
What wastes ten minutes
Looking at a dashboard showing plenty of idle CPU and concluding the scheduler is wrong. It is not wrong, it is answering a different question, and the difference between requests and usage is the whole of it.
XV
Memoria Deest
Insufficient Memory
There is room in the house, but not in one room, and you asked for one room.
What it isThe scheduler found no single node with enough free memory to satisfy your request. Note the word request. Limits are not consulted here. A Pod requesting 8Gi is unschedulable on a cluster of 4Gi nodes no matter how idle they are.
Look here firstkubectl describe node | grep -A 5 'Allocated resources' to see what is already promised, which is not the same as what is in use.
What wastes ten minutes
Looking at a monitoring dashboard showing plenty of free memory. The scheduler does not schedule against usage. It schedules against requests.
XVI
Sine Loco
PodUnschedulable
Every door was tried. Every door had a reason.
What it isReported as 0/N nodes are available. The scheduler examined every node and rejected all of them, and the message tells you why in a compressed form worth learning to read: how many failed for taints, how many for resources, how many for affinity, how many for volume topology.
Look here firstRead the counts in the message before anything else. 3 node(s) had untolerated taint and 3 Insufficient cpu are different problems with the same symptom.
What wastes ten minutes
Adding nodes when the reason was a taint or a node selector. The new nodes will be rejected for the same reason, and now it costs more.
XVII
Initium Impossibile
RunContainerError
The vessel was made ready, and it will not start.
What it isThe runtime accepted the container and then failed to run it. The config was fine, which distinguishes this from its neighbour: the problem is the command, the entrypoint, a mount that cannot be made, or a permission on something the process needs before it has run a single line.
Look here firstkubectl describe pod <pod> and read the message after RunContainerError:. It is one of the more specific messages in the system and it usually names the file.
What wastes ten minutes
Assuming the image is broken and rebuilding it. The image is generally fine. What is wrong is what you asked the image to do on this particular node.
XVIII
Configuratio Corrupta
CreateContainerConfigError
It is written, and what it refers to is not.
What it isThe kubelet cannot even assemble the container, because something the Pod references does not exist: a ConfigMap, a Secret, or a single key inside one. A missing key produces exactly the same status as a missing object.
Look here firstkubectl describe pod <pod> names the missing thing exactly. Then kubectl get secret <name> -o jsonpath='{.data}', because the object existing and the key existing are two different facts.
What wastes ten minutes
Confirming the Secret exists and stopping there. That is the check that feels like the answer and is not, roughly half the time.
XIX
Terminus Transgreditus
DeadlineExceeded
The measure was set before you began, and you have passed it.
What it isA Job ran longer than its activeDeadlineSeconds and was stopped. The work was not necessarily failing. It was merely still going, which the deadline does not distinguish between.
Look here firstkubectl get job <name> -o jsonpath='{.spec.activeDeadlineSeconds}', then compare it with how long the job has actually been taking lately rather than how long it took when the number was chosen.
What wastes ten minutes
Raising the deadline without looking at the trend. If the runtime is growing, you have bought one more run and scheduled the same page for next month.
XX
Ignotum
Unknown
There will be an error that no chant names. It will come on a Friday.
What it isThe status could not be determined, which is the cluster being honest. And more broadly: every list of errors is a list of the errors somebody already had. The twentieth chant is not an error, it is a method, and it is the only entry here that keeps working when the other nineteen do not apply.
Look here firstIn order: kubectl get events -A --sort-by=.lastTimestamp | tail -30, then kubectl get nodes, then the logs of the controller that owns the object. Events first, because they are the only place the cluster writes down what it tried.
What wastes ten minutes
Searching the exact error string on the internet before reading the events in your own cluster. Your cluster knows what happened. The internet knows what happened to somebody else.