kubectl debug Guide: How to Debug Any Kubernetes Pod (2026)

Earlier, when I worked with normal virtual machines (VMs), debugging was simple. I would SSH into the machine, check logs, run commands like top, install tools using apt-get, and fix the issue. Everything was easy and available.

At first, Kubernetes felt similar. Most containers were built using full Linux systems like Ubuntu, Debian, or Alpine. They had shells and package managers. You could run kubectl exec and debug easily.

But things changed when teams started using distroless and scratch images. These images are smaller, faster, and more secure because they don’t include extra tools. But they also remove everything unnecessary, including debugging tools.

Then one day, a Pod breaks at 2 AM.

kubectl exec -it pod-name -- bash
OCI runtime exec failed: exec failed: unable to start container process:
exec: "bash": executable file not found in $PATH

No bash. No shell. No curl. No tools at all. Just your app binary.

This is great for security, but very hard for debugging.

I’ve seen teams waste 30–40 minutes rebuilding images with debug tools, pushing them, and redeploying - only to find the issue is gone or hidden.

There's a much better way to handle this, and it's been stable in Kubernetes since v1.25: kubectl debug.

Let’s go through real scenarios, commands, and important tips.

‍We're putting together an ebook called "The Ultimate Kubernetes Troubleshooting Guide" that goes way beyond what I can fit in a blog post- dozens of failure scenarios across Pods, networking, storage, and cluster infrastructure. More on that at the end.

What's Actually Happening When You Run kubectl debug?

Before we get into the scenarios, it's worth understanding the mechanics. It'll save you confusion later.

kubectl debug does not restart your Pod. It doesn't replace any of your containers. What it does is patch the Pod spec by adding an entry to a field called ephemeralContainers. That's it. A new container starts up next to your application container and you get a shell in it.

Here's the Pod spec before you run anything:

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: myapp
    image: gcr.io/distroless/static:latest
    ports:
    - containerPort: 8080
status:
  phase: Running

Now run:

kubectl debug pod/myapp -it --image=busybox --target=myapp

And the spec becomes:

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: myapp
    image: gcr.io/distroless/static:latest
    ports:
    - containerPort: 8080
  ephemeralContainers:
  - name: debugger-abc12
    image: busybox
    stdin: true
    tty: true
    targetContainerName: myapp
status:
  phase: Running
  ephemeralContainerStatuses:
  - name: debugger-abc12
    state:
      running:
        startedAt: "2026-05-04T10:00:00Z"

That new ephemeralContainers field is the whole trick. The container shares the Pod's network namespace, IPC namespace, and IP address. It can see mounted volumes. But it can't define ports, resource requests, liveness probes, or any of that. And if it exits, Kubernetes won't restart it. These limits are by design - the thing is meant for inspection only.

When to use --copy-to instead

There are situations where attaching an ephemeral container isn't enough. The most common one: your Pod is crashing too fast for you to look around. You need the container to actually stay alive while you poke at its filesystem and environment.

That's what --copy-to is for. It creates a copy of your Pod (and lets you swap the image or override the entrypoint) so you can debug a near-identical version without touching the original.

kubectl debug pod/myapp -it --copy-to=debug-myapp --container=myapp -- /bin/sh

Use ephemeral containers (no --copy-to) when:

• The Pod is running and you want to inspect what's happening live.
• You need the exact same network namespace as the running app.
• You don't want to disturb the original Pod at all.

Use --copy-to when:

• The container is in CrashLoopBackOff and you need it to stay alive.
• You want to override the entrypoint to skip whatever's failing on startup.
• You want to swap to an image with proper debug tools while keeping the same volumes and config.

One thing to remember: a --copy-to Pod is a separate Pod. It runs on the same node by default, but it's not your real workload, and Kubernetes won't clean it up for you. Delete it when you're done.

Scenario 1: Your Pod has no shell

This one comes up the most. A team ships a distroless or scratch image (which they should), and a few weeks later someone needs to debug something and finds they can't exec in.

Don't rebuild the image. Attach a debug container:

kubectl debug pod/myapp -it --image=busybox --target=myapp

The --target flag matters. Without it, the debug container gets its own process namespace, so ps aux only shows processes inside the debug container. Not useful. With --target=myapp, the debug container shares the app container's process namespace and you can actually see what the app is doing.

From there you've got the full Pod network stack. Test DNS, curl an endpoint, check whether a ConfigMap mounted properly, read a Secret file. Whatever you need - without touching the running app.

Scenario 2: Network problems and no tools

Network issues between services are some of the most annoying things to chase. Symptoms are vague. Timeouts come and go. DNS fails sometimes. One Pod can reach a service, another can't. And of course your production image doesn't have curl or dig or tcpdump - why would it?

Pick a debug image that fits. Skip BusyBox for network stuff and go straight to Netshoot:

kubectl debug pod/api-server -it --image=nicolaka/netshoot

Netshoot ships with pretty much every network tool you'd want. Once you're in:

# Does the service name resolve?
dig backend-service.default.svc.cluster.local

# Can we reach it?
curl -v http://backend-service:8080/health

# Where does traffic go?
traceroute backend-service

# Look at the raw packets
tcpdump -i eth0 -n port 8080

Because the debug container shares the network namespace with your app, what you see is exactly what the app sees. If dig returns NXDOMAIN, that's the same DNS failure your app is hitting. There's no "well it works from my test Pod" problem - the network context is identical.

That alone saves a lot of time vs. spinning up a separate test Pod and trying to recreate the conditions manually.

Scenario 3: Pod stuck in CrashLoopBackOff

This is the worst one. Container starts, crashes, Kubernetes restarts it, it crashes again, the backoff grows, and you can't exec in because the container is never alive long enough.

Start with the obvious stuff:

# Events - scheduling? image pull? OOM?
kubectl describe pod myapp

# Logs from the container that just died
kubectl logs myapp --previous

--previous pulls logs from the last terminated instance. Most of the time the crash reason is right there - a bad env var, a wrong DB connection string, a panic on startup.

But sometimes logs don't tell the full story. App crashes before it logs anything. Or the issue is a missing config file or a volume that didn't mount. You need to look at the container's environment as it would have started.

This is where --copy-to earns its keep:

kubectl debug pod/myapp -it --copy-to=debug-myapp --container=myapp -- /bin/sh

This creates a copy of the Pod, but instead of running the app binary, it drops you into /bin/sh. Same image, same volumes, same env vars - except now nothing is crashing and you can take your time.

Once inside, the usual checks:

# Env vars
env

# Config file there? Looks right?
ls -la /etc/config/
cat /etc/config/app.yaml

# Did the secret volume mount?
ls -la /var/run/secrets/

# File permissions
ls -la /app/

I reach for --copy-to whenever the crash happens during startup and I need to see what the container filesystem actually looks like before the binary tries to run.

When you're done, don't forget to clean up the copy:

kubectl delete pod debug-myapp

Scenario 4: The problem is the node, not the Pod

Most people think kubectl debug is just for Pods. That's the common case. But it works on nodes too.

A lot of issues aren't your application's fault. Disk pressure on the node. A misbehaving kubelet. Container runtime weirdness. Host-level networking. For these, you need to inspect the node itself.

kubectl debug node/worker-node-1 -it --image=ubuntu

This creates a privileged Pod on that specific node and mounts the node's root filesystem at /host. From there:

# Disk
df -h /host

# Recent kubelet logs
journalctl -u kubelet --no-pager -n 50 --root /host

# Containers running on this node
chroot /host crictl ps

# Network interfaces
chroot /host ip addr show

On managed clusters (GKE, EKS, AKS) you usually don't have easy SSH to nodes. This gets you the same level of inspection without bastion hosts or asking the platform team for access.

Best Practices when using kubectl debug

Here are the best practices when you are using kubectl debug:

1. Match the image to the problem: BusyBox is fine for filesystem stuff. Network problems? Go straight to Netshoot. Need a full-blown Linux environment? Ubuntu or Alpine. Don't waste time installing curl in a BusyBox container when you could've just used the right image from the start.

2. Look before you touch: I know it's tempting to start changing things when you're under pressure, but resist that urge. Run env. Run ps. Run ls. Run cat on the config files. Get the lay of the land before you change anything in a production environment. The last thing you need is to turn a debugging session into a second incident.

3. Don't leave debug containers hanging around: Here's the thing - Kubernetes won't let you remove an ephemeral container from a Pod. Once it's there, it stays until the Pod gets deleted or replaced. If you attached an Ubuntu image to a Pod that normally runs a 15MB distroless container, you're now burning extra memory for no reason.

‍4. Keep the application container untouched: The whole point of kubectl debug is to avoid messing with your production image. If you catch yourself thinking, "I'll just apt-get install tcpdump inside the app container real quick," stop. Attach a debug container. Keep the immutability principle intact.

When kubectl Isn't enough?

 Not every issue calls for an ephemeral container. Plenty of times, simpler tools will get you there faster.

kubectl describe pod myapp should honestly be your first command for almost any Pod issue. The Events section at the bottom tells you about image pull errors, failed mounts, scheduling failures, OOM kills - a ton of useful context that doesn't require a debug session.

kubectl logs myapp (and kubectl logs myapp --previous) covers most application-level crashes. If the app logged the error, you'll find it here.

If you need to tail logs from multiple Pods at once, common with replicated services,  stern is great. And for navigating cluster resources interactively, k9s gives you a terminal dashboard that's way faster than typing out kubectl commands all day.

Save kubectl debug for when you've already checked logs and events and you need to go deeper. That's where it shines: containers with no shell, network layer problems, and node-level issues. The stuff where you really do need to get your hands dirty.

Where to Go From Here

I covered the four scenarios I run into most often, but there's a lot more that can go sideways in a production Kubernetes cluster. Scheduling conflicts. PVC binding failures. DNS misconfigurations at the cluster level. Node pressure evictions. Resource quota headaches.

We dig into all of those in our ebook, "The Ultimate Kubernetes Troubleshooting Guide." It's got step-by-step workflows for dozens of real failure scenarios – pods, networking, storage, nodes, the whole stack. If you want one reference that covers everything your cluster might throw at you, this guide has you covered.