.jpg)
Subscribe to our newsletter
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
.png)
.png)
What Is Kubernetes Troubleshooting?
Kubernetes troubleshooting is the systematic process of identifying and resolving issues across pods, services, and clusters to maintain application health. The process generally follows a "bottom-up" approach, starting with the smallest unit, the pod, and scaling up to cluster-wide networking and nodes.
The Kubernetes troubleshooting process relies on a combination of command-line tools, log inspection, event analysis, and interactive debugging sessions. Effective troubleshooting demands familiarity with tools like kubectl, as well as knowledge of common error patterns and their solutions. By following structured troubleshooting workflows, engineers can reduce downtime, prevent cascading failures, and maintain application reliability.
A Kubernetes troubleshooting workflow typically includes these steps:
- Check pod status: Run
kubectl get podsto identify failing pods. - Inspect events: Use
kubectl describe pod <pod-name>to see a history of events (e.g., scheduling failures or image pull errors). - View logs: Execute
kubectl logs <pod-name>to see application-level errors. - Interactive debugging: Use kubectl
exec -it <pod-name> -- /bin/shto test internal connectivity or inspect configuration files.
Common error codes and fixes include:
In this article:
- Why Is Kubernetes Troubleshooting Important?
- Core Kubernetes Troubleshooting Workflow
- Essential kubectl Commands for Troubleshooting
- Common Kubernetes Error Codes and Fixes
- 5 Ways to Prevent Kubernetes Issues in the First Place
Why Is Kubernetes Troubleshooting Important?
Kubernetes troubleshooting plays a critical role in maintaining stable and reliable containerized environments. Kubernetes clusters contain many interconnected components, including pods, nodes, services, ingress controllers, storage systems, and networking layers.
A failure in one component can quickly affect other parts of the system. Troubleshooting helps engineers identify issues early, isolate root causes, and restore normal operations with minimal disruption:
- Reduces application downtime: Kubernetes workloads can fail for many reasons, including container crashes, failed deployments, networking problems, or node outages. Troubleshooting helps teams locate the source of the issue and restore affected services.
- Maintains high availability: Production systems often rely on Kubernetes to run business applications. Troubleshooting helps ensure that applications remain accessible and responsive when infrastructure components fail or workloads become unstable.
- Identifies configuration and deployment errors: Many Kubernetes issues are caused by incorrect YAML configurations, invalid environment variables, resource limits, or service definitions. Troubleshooting helps detect these mistakes and verify that workloads are deployed correctly.
- Prevents cascading failures: Problems in Kubernetes environments can spread quickly. For example, resource exhaustion on one node may affect scheduling across the cluster. Troubleshooting helps isolate failures before they impact additional workloads or services.
- Improves performance and resource efficiency: Troubleshooting helps identify CPU throttling, memory pressure, disk bottlenecks, and network latency issues. Resolving these problems improves workload performance and ensures cluster resources are used efficiently.
- Speeds up incident response: Structured troubleshooting workflows help engineers respond to incidents more efficiently. Using logs, events, metrics, and debugging tools helps teams reduce mean time to detection (MTTD) and mean time to recovery (MTTR).
- Supports reliable scaling and updates: Kubernetes environments frequently change due to autoscaling, rolling updates, node replacements, and configuration updates. Troubleshooting helps validate that these changes do not introduce instability or unexpected behavior.
Core Kubernetes Troubleshooting Workflow
1. Check Pod Status
The first step in troubleshooting a Kubernetes issue is to check the status of relevant pods using kubectl get pods or kubectl describe pod <pod-name>. This provides information about the pod’s phase, readiness, restarts, and error messages. Reviewing the pod’s status helps identify if it is running, pending, failed, or stuck in a crash loop.
Detailed pod descriptions reveal recent state transitions, container statuses, and reasons for failures or restarts. These details can point to misconfigurations, image pull failures, resource constraints, or application-level errors. Checking pod status early in the workflow helps focus on the right components.
Example
kubectl get pods
kubectl describe pod nginx-7d8b49557c-xk9m2
2. Inspect Events
Kubernetes events provide a chronological log of significant occurrences within the cluster, such as pod scheduling failures, image pull errors, or resource quota violations. Using kubectl get events or kubectl describe <resource> helps surface recent events related to the problematic resource. These events often include error codes and descriptive messages.
Events are ephemeral and may be overwritten quickly in busy clusters, so review them promptly. Correlating event timestamps with incident reports or monitoring alerts can help reconstruct the sequence of events leading to an issue. Consistent inspection of events supports root cause analysis and reduces the risk of overlooking critical signals during incident response.
Example
kubectl get events --sort-by=.metadata.creationTimestamp
kubectl describe pod nginx-7d8b49557c-xk9m2
3. View Logs
Application and system logs help you understand container behavior and identify the root cause of failures. The kubectl logs <pod-name> command retrieves logs for a specific container, showing stack traces, error messages, and application output. Reviewing logs helps distinguish between infrastructure-level and application-level problems.
For pods with multiple containers or those that have crashed, flags like -c <container-name> or --previous provide access to specific logs. Log analysis shows what the application was doing before the failure, highlighting configuration errors, code bugs, or resource exhaustion.
Example
kubectl logs nginx-7d8b49557c-xk9m2
kubectl logs nginx-7d8b49557c-xk9m2 -c app --previous
4. Interactive Debugging
When status checks, events, and logs do not provide enough context, interactive debugging may be required. Tools like kubectl exec -it <pod-name> -- /bin/sh allow engineers to open a shell inside a running container for real-time inspection. This enables execution of troubleshooting commands, examination of the file system, and verification of network connectivity from within the pod.
Interactive debugging is useful for investigating environment variables, mounted volumes, or application state that may not be visible from outside the container. It also supports testing connectivity to external services or reproducing application errors. Use interactive debugging carefully in production environments to avoid unintended side effects.
Example
kubectl exec -it nginx-7d8b49557c-xk9m2 -- /bin/sh
Essential kubectl Commands for Troubleshooting
Pod and Deployment Commands
The kubectl get pods command provides an overview of pod states, readiness, restart counts, and node assignments. It is used to identify pods that are stuck in Pending, CrashLoopBackOff, ImagePullBackOff, or other abnormal states. Adding flags like -o wide displays details such as node placement and pod IP addresses.
The kubectl describe pod <pod-name> command displays detailed information about a pod, including container states, mounted volumes, environment variables, and recent events. This command helps identify scheduling failures, resource issues, and container startup errors.
To inspect deployment health, engineers use kubectl get deployments and kubectl describe deployment <deployment-name>. These commands show rollout status, replica counts, update strategies, and deployment events. They help determine whether pods are failing because of rollout issues or invalid deployment configurations.
The kubectl rollout status deployment/<deployment-name> command monitors deployment progress in real time. If a rollout becomes stuck or fails, kubectl rollout undo deployment/<deployment-name> reverts the deployment to the previous stable revision. These commands are used during failed application updates or configuration changes.
Related content: Keep our kubectl cheat sheet handy for a full reference of troubleshooting commands.
Node Commands
Node-level issues can affect scheduling, networking, and workload stability across the cluster. The kubectl get nodes command shows node readiness and high-level status information. Nodes marked as NotReady or SchedulingDisabled often indicate infrastructure or kubelet-related problems.
The kubectl describe node <node-name> command provides node diagnostics, including resource capacity, allocatable resources, running pods, taints, conditions, and recent events. This information helps identify CPU pressure, memory exhaustion, disk pressure, or networking issues affecting workloads on the node.
The kubectl top nodes command displays node resource consumption, including CPU and memory usage. High utilization may explain pod evictions, throttling, or scheduling failures. Combining this with kubectl top pods helps correlate node resource pressure with problematic workloads.
For deeper investigation, engineers may inspect kubelet logs or system services directly on the node using tools such as journalctl. Node troubleshooting is important when pods fail across multiple namespaces or when cluster-wide performance degradation occurs.
Service and Networking Commands
Networking issues in Kubernetes often involve services, DNS resolution, ingress rules, or network policies. The kubectl get services command lists cluster services and their assigned IP addresses, ports, and types. This helps verify whether services are exposed correctly and mapped to the expected workloads.
The kubectl describe service <service-name> command shows service selectors, endpoints, and event information. Missing endpoints usually indicate that pods do not match the service selector labels or are failing readiness checks.
The kubectl get endpoints command verifies which pod IPs are attached to a service. If no endpoints appear, traffic cannot reach backend pods even if the service exists.
DNS and connectivity issues are often investigated using temporary debug containers or interactive shells. Commands like kubectl exec -it <pod-name> -- nslookup <service-name> or curl <service-name> help verify internal DNS resolution and service reachability. These checks help isolate whether failures originate from DNS configuration, network policies, ingress rules, or application-level connectivity problems.
Common Kubernetes Error Codes and Fixes
ImagePullBackOff
Issue:ImagePullBackOff occurs when Kubernetes cannot pull a container image required to start a pod. After the initial image pull failure, Kubernetes retries the operation with progressively longer delays. Until the image becomes available, the pod remains in a non-running state.
Diagnosis:
Image pull failures are commonly caused by invalid image names, incorrect tags, missing registry credentials, expired authentication tokens, registry outages, or network connectivity problems. The most useful starting point is the Events section of the pod description.
Run kubectl describe pod <pod-name> and look for messages such as ErrImagePull, pull access denied, unauthorized, or manifest unknown. These messages usually indicate whether the failure is related to authentication, image availability, or connectivity.
Resolution Workflow:
- Run
kubectl describe pod <pod-name>and review the Events section. - Verify that the image name and tag are correct and exist in the registry.
- Confirm that the container registry is reachable from cluster nodes.
- Validate image pull secrets configured on the pod or service account.
- Check registry credentials, permissions, and token expiration status.
- Verify DNS resolution and outbound network connectivity from worker nodes.
- Review proxy, firewall, or network policy configurations that may block registry access.
Example
kubectl describe pod my-pod
kubectl get secrets
kubectl describe secret registry-secret
CrashLoopBackOff
Issue:CrashLoopBackOff indicates that a container repeatedly starts, exits, and is restarted by Kubernetes. After several consecutive failures, Kubernetes introduces increasing delays between restart attempts to avoid constant restart cycles.
Diagnosis:
This condition is typically caused by application startup failures, invalid configuration values, missing environment variables, failed dependency initialization, database connectivity problems, or incorrect container commands.
Container logs are usually the fastest way to identify the root cause. Review logs using kubectl logs <pod-name>. If the container restarts too quickly, use the --previous flag to retrieve logs from the last failed instance. The pod description can also reveal termination reasons, exit codes, restart counts, and probe failures.
Resolution Workflow:
- Review application logs using
kubectl logs <pod-name>. - If necessary, retrieve logs from the previous container instance using
--previous. - Inspect pod events and termination details with
kubectl describe pod <pod-name>. - Verify environment variables, configuration files, secrets, and startup commands.
- Check database connections, external dependencies, and service endpoints.
- Review CPU and memory limits for signs of resource exhaustion.
- Validate readiness, startup, and liveness probe settings and adjust timing thresholds if required.
Example
kubectl logs my-pod
kubectl logs my-pod --previous
kubectl describe pod my-pod
Pending
Issue:
A pod enters the Pending state when Kubernetes accepts the pod definition but cannot assign it to a suitable node. The workload remains unscheduled and does not start until all scheduling requirements are satisfied.
Diagnosis:
Pending pods are most often caused by insufficient cluster resources, restrictive scheduling rules, storage provisioning failures, or node placement constraints. Common scheduling messages include insufficient CPU, insufficient memory, unmatched node selectors, taint conflicts, or unbound PersistentVolumeClaims.
The scheduling events displayed by kubectl describe pod <pod-name> typically identify the exact reason the scheduler could not place the workload.
Resolution Workflow:
- Run
kubectl describe pod <pod-name>and review scheduler events. - Check node availability using
kubectl get nodes. - Verify that sufficient CPU and memory resources exist in the cluster.
- Review node selectors, node affinity rules, taints, and tolerations.
- Validate that required node labels are present.
- Check
PersistentVolumeClaims(PVC) and storage provisioning status. - If autoscaling is enabled, confirm that the cluster autoscaler is functioning correctly.
Example
kubectl describe pod my-pod
kubectl get nodes
kubectl get pvc
OOMKilled
Issue:OOMKilled occurs when a container exceeds its configured memory limit and is terminated by the Linux out-of-memory (OOM) killer. Kubernetes enforces memory limits strictly, causing the container to stop immediately when the limit is exceeded.
Diagnosis:
Memory exhaustion can result from memory leaks, excessive caching, inefficient application behavior, traffic spikes, large data sets, or memory limits that are set too low for the workload.
The pod description usually shows OOMKilled as the termination reason. Resource monitoring tools such as Prometheus, Grafana, or Metrics Server can help determine whether the workload experiences sustained memory pressure or sudden usage spikes.
Resolution Workflow:
- Confirm the termination reason using
kubectl describe pod <pod-name>. - Review memory consumption metrics before the failure occurred.
- Identify memory spikes, leaks, or sustained high utilization.
- Compare actual memory usage against configured requests and limits.
- Increase memory limits if they are clearly undersized.
- Investigate application-level causes such as caching, concurrency, or memory leaks.
- Optimize application memory usage and adjust resource settings based on observed workload behavior.
Example
kubectl describe pod my-pod
kubectl top pod my-pod
kubectl top nodes
5 Ways to Prevent Kubernetes Issues in the First Place
Smart Kubernetes teams plan their clusters in a way that minimizes common faults. This supports resilience and also enables easier troubleshooting when issues do arise.
1. Set Resource Requests and Limits
Resource requests and limits help Kubernetes allocate CPU and memory predictably across the cluster. Requests define the minimum resources required for scheduling, while limits define the maximum resources a container can consume. Proper configuration prevents workloads from competing aggressively for shared resources.
Without requests, the scheduler cannot make accurate placement decisions, which may lead to overloaded nodes and unstable workloads. Without limits, a single container may consume excessive memory or CPU, affecting other applications running on the same node.
Carefully configured limits help reduce the risk of OOMKilled events, CPU throttling, and node resource exhaustion. However, limits that are set too low can also create instability by restricting applications during peak load periods.
Using LimitRange and ResourceQuota policies at the namespace level helps enforce consistent resource management practices across teams and environments. These controls reduce the likelihood of misconfigured workloads affecting cluster stability.
2. Use Readiness and Liveness Probes Carefully
Readiness and liveness probes help Kubernetes determine whether containers are healthy and ready to receive traffic. Proper probe configuration improves application reliability by preventing unhealthy pods from serving requests and automatically restarting failed containers.
Readiness probes control whether a pod is added to a service endpoint list. If the readiness probe fails, traffic stops routing to the pod until it recovers. This helps prevent users from reaching partially initialized or degraded applications.
Liveness probes detect application deadlocks or unresponsive processes. When a liveness probe fails repeatedly, Kubernetes restarts the container automatically. This mechanism improves self-healing behavior but can also create instability if configured incorrectly.
Aggressive probe timing is a common source of unnecessary restarts and CrashLoopBackOff conditions. Applications that require long startup times may fail health checks before initialization completes. Startup probes are often useful for workloads with slow boot processes.
3. Monitor Events, Logs, and Metrics
Continuous monitoring is essential for detecting Kubernetes issues before they escalate into outages. Logs, metrics, and cluster events provide visibility into workload behavior, resource usage, infrastructure health, and operational failures.
Metrics platforms such as Prometheus and Grafana help track CPU usage, memory consumption, pod restarts, network latency, and storage performance. Monitoring trends over time allows engineers to identify abnormal behavior before services become unstable.
Centralized logging systems improve troubleshooting by aggregating container logs, node logs, and system events into a searchable platform. Tools such as Elasticsearch, Fluent Bit, Loki, and Splunk are commonly used for Kubernetes log collection and analysis.
Kubernetes events provide additional operational context by recording scheduling failures, image pull errors, pod evictions, and configuration issues. Because events are temporary, collecting and storing them externally improves long-term troubleshooting capabilities.
4. Use Namespaces and Labels Consistently
Namespaces help organize workloads and isolate resources within a Kubernetes cluster. Consistent namespace usage improves operational clarity, access control, resource management, and troubleshooting efficiency.
Separating environments such as development, staging, and production into different namespaces reduces the risk of accidental interference between workloads. Namespace-level policies can also enforce security boundaries and resource quotas.
Labels provide metadata used for workload selection, grouping, monitoring, and automation. Kubernetes services, deployments, network policies, and monitoring systems all rely heavily on labels for workload identification.
Inconsistent or poorly designed labeling schemes often cause service routing failures, deployment confusion, and operational complexity. Standardized labels make it easier to search, monitor, and manage workloads across large clusters.
5. Continuously Right-Size Workloads Based on Real Usage
Workload resource requirements change over time as applications evolve, traffic patterns shift, and infrastructure usage grows. Continuously right-sizing workloads helps maintain performance while reducing wasted cluster resources.
Overprovisioned workloads waste CPU and memory, increasing infrastructure costs and reducing cluster efficiency. Underprovisioned workloads create instability, scheduling failures, CPU throttling, and memory-related crashes.
Monitoring actual workload behavior provides the data needed to optimize resource requests and limits. Engineers should review long-term usage patterns instead of relying only on short-term spikes or initial deployment estimates.
Tools such as Vertical Pod Autoscaler (VPA) can recommend or automatically adjust resource settings based on observed consumption. Horizontal Pod Autoscaler (HPA) complements this approach by scaling replica counts in response to load changes.
How PerfectScale Helps You Prevent and Resolve Kubernetes Issues
Most Kubernetes troubleshooting is reactive: you wait for a pod to crash, then dig through events, logs, and metrics to find the cause. PerfectScale flips that model by autonomously right-sizing workloads, preventing downtime, and optimizing resource use to maintain 99.99% availability. Instead of manually chasing OOMKilled, CrashLoopBackOff, and throttling events, your DevOps, SRE, and platform engineering teams get continuous, data-driven remediation across every layer of the K8s stack.
Key capabilities of PerfectScale:
- Automatic issue remediation: Instantly identify and fix resiliency risks to maximize uptime and eliminate latency, addressing problems such as OOM, CPU throttling, and pod evictions before they cascade into outages.
- Configuration error prevention: Catch the misconfigurations behind common failures, including missing CPU requests, missing memory requests, and unset memory limits, so workloads start and run reliably.
- Under-provisioning detection: Surface under-provisioned CPU and memory (both requests and limits) and suspected memory leaks, the root causes that keep pods stuck in restart loops even when nodes look healthy.
- Infrastructure hardening: Get holistic visibility across nodes to prevent node over-commitment, validate node affinities and taints, and choose the optimal node types for your workloads.
- Impact-driven prioritization: Align alerting with your SLAs and SLOs, receive instant notifications through Slack, MS Teams, or Datadog, and escalate issues into a ticket with one click.
- Autoscaling optimization: Fine-tune horizontal and vertical autoscaling configurations so scaling triggers fire accurately, preventing the "HPA at max replicas" and resource-exhaustion scenarios that drive instability.
Ready to stop firefighting Kubernetes issues? Explore PerfectScale's Kubernetes performance and resilience platform to see how autonomous optimization keeps your clusters stable, resilient, and cost-efficient.
