kubectl logs: How to View and Debug Pod Logs in Kubernetes

kubectl logs lets you read the logs of a container running in a pod. The container runtimes capture what the application writes to standard output (stdout) and standard error (stderr) and make those logs available through the kubelet. kubectl logs is the usual way to access them.

You don't need to SSH into a node and search through log files just to read a container's logs.

In this guide, you'll learn how Kubernetes stores and provides container logs, the important kubectl logs flags, how to get logs from individual pods and larger workloads, how to follow and filter log output, how to read logs from a previous container instance, and how to access node-level logs. Just as importantly, you'll learn about the failures that kubectl logs cannot show you.

Searching and parsing kubectl logs output

kubectl logs prints plain text, so you can pipe it to other commands. To show only lines that contain "error", or to save the output for later:

kubectl logs <pod> | grep -i error
kubectl logs <pod> > pod.log

If your application logs in JSON, jq lets you query the output instead of reading it line by line. For example, to show only error-level entries and print one field from each:

kubectl logs <pod> | jq 'select(.level=="error") | .msg'

This is the main reason to log in a structured format.

Querying node and kubelet logs with kubectl logs

The Pod logs do not cover the node's own services, such as the kubelet and the container runtime. The Node log query feature lets you read those through the API without connecting to the node directly. It was introduced in Kubernetes 1.27, went to beta in 1.30, and is GA as of 1.36. On 1.36 and later the NodeLogQuery feature gate is enabled by default; you still set the kubelet options enableSystemLogHandler and enableSystemLogQuery to true. On older clusters, enable the NodeLogQuery feature gate as well. Once it is enabled, you query node logs with kubectl get --raw:

# kubelet logs from a node
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/logs/?query=kubelet"

# container runtime logs
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/logs/?query=containerd"

# filter for a keyword
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/logs/?query=kubelet&pattern=error"

On Linux, the kubelet retrieves these logs using journalctl, and on Windows it reads the event log. In order to read node logs, it requires get and create access to nodes/proxy in RBAC.

Node logs matter when the problem is below the pod. A common example is a node that reaches the Ready state, but pods on it never start because of a CNI (networking) misconfiguration. The pod has no logs yet, so kubectl logs shows nothing, and the real error is in the kubelet log. Kubelet or runtime crashes, disk-pressure evictions, and image-mount failures also appear here rather than in a pod's output.

Resource problems that kubectl logs will not show

Many pod failures leave no log at all, because they happen to the container rather than inside it. kubectl logs shows what your application printed, and an application that is killed or throttled usually prints nothing useful.

The clearest example is an OOMKilled container. When a container uses more memory than its limit, the kernel kills it, and the application does not get a chance to log anything. The kubectl logs and kubectl logs -p may be empty or incomplete. In this case, check the pod's state instead:

kubectl describe pod <pod>

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Exit code 137 means the container was killed with SIGKILL, which usually means it ran out of memory. kubectl logs does not show this; the pod's state does.

CPU throttling is different. When a container reaches its CPU limit, the kernel slows it down instead of stopping it. The container keeps running, so there may be no error in the logs. You may instead see higher p99 latency without seeing more errors in the logs. Metrics such as container_cpu_cfs_throttled_periods_total can show the throttling.

Due to the above cases, you have to look at the pod's state and events whenever the logs are empty or do not explain a failure. kubectl describe pod <pod> shows the restart count, the last exit code, and recent events. kubectl events, or kubectl get events --sort-by=.lastTimestamp, shows what Kubernetes did, such as scheduling, image pulls, evictions, and OOM kills. Logs and events together give the full picture.

There is also a cost to streaming logs at scale. Every kubectl logs request goes through the API server to the kubelet, and --follow keeps the connection open. Streaming logs from hundreds of pods or polling them frequently can put extra load on the API server. This is why --max-log-requests defaults to 5 and why it helps to limit requests with --since and --tail.

Tools that extend Kubernetes log and resource visibility

The following tools help with resource issues, multiple pods, and long-term log storage:

PerfectScale

PerfectScale helps with failures that do not appear in application logs. Many pod restarts are caused by resource problems, such as OOMKilled or CPU throttling, rather than application bugs. These problems may not leave anything useful in the logs. PerfectScale's Kubernetes governance platform watches how your workloads actually use CPU and memory, identifies these resource-caused failures, and turns them into actionable, automated right-sizing recommendations that you can apply manually or autonomously. Pods then stop being killed and throttled in the first place, which means fewer restarts to investigate and fewer empty crash logs to read. Teams like Paramount Pictures and Creditas use PerfectScale to optimize their clusters. You can sign up or book a technical session.

Stern

Stern tails logs from many pods at once, with color-coded output. It is easier to work with than kubectl logs -l when you are watching several pods during a rollout.

Grafana Loki

Grafana Loki stores logs centrally and makes them searchable. It lets you read logs from a pod that ran last week, which kubectl logs cannot do.

Fluent Bit

Fluent Bit is a lightweight log collector, usually run as a DaemonSet. It reads each node's container logs and ships them to a store such as Loki or Elasticsearch before rotation deletes them.

Best practices for using kubectl logs

Here are some best practices to follow when using kubectl logs:

1. Write application logs to stdout and stderr

You have to avoid writing logs to files inside the container and use a structured format such as JSON, which also makes logs easier to search and process with tools like jq and Loki.

2. Scope every request with --since and --tail before streaming

You should avoid printing a pod's entire history when you only need the last few minutes. It is faster for you and puts less load on the API server.

3. Check the pod status before reading its logs

You should use kubectl get pod and kubectl describe pod to check restarts and exit codes. For example, an OOMKilled container may not have useful logs.

4. Send logs to a central store

The logs are stored on the node and can be removed during rotation or when the pod is deleted. A central logging system keeps them available for longer.

5. Restrict access to the pods/log subresource with RBAC

Users need access to the pods/log resource to view pod logs. Give this permission only where it is needed, because logs may contain sensitive information.

Troubleshooting common kubectl logs errors

Here are some common kubectl logs errors and how to troubleshoot them:

Multiple containers in the pod

If a pod has more than one container, specify the container with -c <container>. To view logs from all containers, use --all-containers=true.

Empty log output

The empty output usually has one of three causes. The application may be writing logs to a file instead of stdout or stderr, you may be checking the wrong container, or the application may not have written any logs yet. You should check where the application writes its logs and confirm the container name.

Kubelet certificate errors

These errors usually mean the API server cannot connect to the kubelet because its serving certificate is expired or not trusted. The proper fix is to repair the certificate. As a temporary workaround, --insecure-skip-tls-verify-backend skips the certificate check, but it should not be used as the permanent solution.