Kubernetes Logs: Types, Commands & 6 Best Practices

What Is Kubernetes Logging?

In Kubernetes, logs provide the primary means of observing application behavior and troubleshooting cluster issues. Because pods are ephemeral, Kubernetes manages logs by capturing stdout and stderr streams from containers and storing them as temporary files on the node.

Logging architecture:

Kubernetes does not provide a native persistent storage solution for logs; it relies on three main architectural patterns:

• Node-level logging: The container runtime captures standard output and stores it in /var/log/pods/ on the host node.
• Sidecar containers: A secondary container inside the pod collects logs from the application and streams them to its own stdout or directly to a logging backend.
• Cluster-level logging: A specialized agent (like Fluentd, Fluent Bit, or Filebeat) runs as a DaemonSet on every node. It collects local log files and forwards them to a centralized storage system like Elasticsearch, Grafana Loki, or cloud-specific services like AWS CloudWatch.

Types of Kubernetes logs:

• Application Logs: The most common type, generated by your code running in pods 5.185.185.18.
• System Component Logs: Generated by core Kubernetes services like the API server, scheduler, and kubelet.
• Audit Logs: Records of every call made to the Kubernetes API server, used primarily for security and compliance.
• Events: Not technically logs, but time-stamped records of state changes in the cluster (e.g., a pod failing to start), viewed via kubectl get events.

In this article:

• Understanding Kubernetes Logging Architecture
• Types of Kubernetes Logs
• Essential kubectl logs Commands
• Common Kubernetes Logging Use Cases
• Kubernetes Logging Best Practices

Understanding Kubernetes Logging Architecture 

Node-Level Logging

Node-level logging in Kubernetes refers to capturing logs at the host machine, where the container runtime stores logs for all containers running on that node. By default, Kubernetes nodes use the container runtime’s logging mechanism, such as Docker’s JSON logging driver, which writes each container’s stdout and stderr streams to log files in a standard location, for example, /var/log/containers/. 

Advantages:

• This approach ensures that logs are persisted on the node even if a container is short-lived or crashes.
• It provides a local source for troubleshooting.

Limitations: 

• It is not advisable to rely solely on node-level logging.
• When nodes are deleted, or if logs are rotated or purged, data can be lost. 
• Node-level logs also remain isolated, making it difficult to correlate events across nodes or aggregate logs for centralized analysis. 

In production environments, organizations typically complement node-level logging with cluster-wide solutions that collect and export logs to external systems for retention, search, and analysis.

Sidecar Containers

Sidecar containers are an architectural pattern in Kubernetes where an additional container runs alongside the primary application container within the same pod. In logging, a sidecar container reads log files or streams generated by the main application container and forwards them to an external logging backend. This design separates logging logic from the application code, allowing teams to standardize log collection across workloads.

‍Advantages: 

• Using sidecar containers for logging offers flexibility and consistency. 
• The sidecar can handle tasks such as log rotation, formatting, enrichment with metadata, and secure transmission to centralized log storage. 
• This approach is useful when applications do not natively support structured logging or when logs need to be shipped to multiple destinations. 

Limitations:

• It introduces resource overhead and operational complexity.
• Each pod must be configured with and manage the lifecycle of its sidecar containers.

Cluster-Level Logging

Cluster-level logging aggregates logs from all nodes, pods, and components in a Kubernetes cluster and exports them to a centralized system. This is typically achieved by deploying log collection agents, such as Fluentd, Logstash, or Filebeat, as DaemonSets, which run on every node and harvest logs from container log files or the journald system. The collected logs are then forwarded to external platforms such as Elasticsearch, Splunk, or cloud-based logging services for storage, indexing, and querying.

Advantages:

• Cluster-level logging addresses the challenges of ephemeral nodes and pods by ensuring that logs are preserved even as workloads are rescheduled or infrastructure changes occur. 
• It enables search, correlation, and analytics across cluster components, making it easier to monitor application health, debug incidents, and meet compliance requirements.
• Implementing cluster-level logging is considered best practice for production Kubernetes environments, as it provides a scalable foundation for observability.

Types of Kubernetes Logs 

1. Application Logs

Application logs are generated by the code running inside containers. These logs capture output from application processes, such as informational messages, errors, warnings, and debugging statements. Developers rely on application logs to understand runtime behavior, trace requests, and diagnose issues related to business logic or third-party dependencies. Application logs are typically written to stdout and stderr, making them accessible to Kubernetes logging mechanisms.

Considerations: 

• Managing application logs in Kubernetes presents challenges due to the dynamic nature of containers. If logs are not collected and persisted outside the container, information can be lost when pods are terminated or rescheduled. 
• Organizations should include emitting logs in a structured format, such as JSON, avoiding log file writes inside containers.
• Centralized logging solutions can retain and search logs for troubleshooting and monitoring.

2. System Component Logs

System component logs are produced by the core infrastructure of a Kubernetes cluster. This includes logs from the kubelet, kube-apiserver, kube-controller-manager, kube-scheduler, and container runtime. These logs provide insight into the operation of the Kubernetes control plane and node agents, capturing events such as scheduling decisions, health checks, authentication attempts, and error conditions within the platform.

Considerations: 

• Accessing and analyzing system component logs is critical for cluster operators and administrators. 
• These logs are often stored on the host filesystem or accessed via systemd journald, depending on the cluster setup. 
• Centralizing system component logs alongside application logs helps correlate infrastructure-level events with application issues, enabling root cause analysis and proactive monitoring of the Kubernetes environment.

3. Audit Logs

Audit logs in Kubernetes record all requests to the API server, capturing details about who performed what action, when, and from where. These logs support security, compliance, and forensic investigations, as they provide a tamper-evident trail of user and system activity. Audit logging is configurable in Kubernetes, allowing administrators to control which events are logged and to what extent, based on policies that define the granularity and retention of audit data.

Considerations: 

• Audit logs differ from application and system logs by focusing on API interactions and access control events. 
• They help organizations detect unauthorized access, monitor sensitive operations, and demonstrate regulatory compliance. 
• Storing audit logs in secure, immutable storage and integrating them with SIEM or security analytics platforms are recommended practices for maintaining security in Kubernetes environments.

Events

Kubernetes events are time-stamped records of significant occurrences within the cluster, such as pod creations, scheduling failures, restarts, or resource quota violations. Events are not traditional logs but serve as notifications that help users understand state transitions and lifecycle changes of Kubernetes objects. Events can be viewed using kubectl get events and are useful for real-time debugging and operational visibility.

Considerations: 

• Events are ephemeral and typically retained for a short duration in the Kubernetes API server, default is one hour. 
• While they provide context for troubleshooting, events should not be used as the sole source of audit or diagnostic information. 
• For broader visibility, events should be collected and exported to external monitoring or logging systems, where they can be correlated with application and system logs.

Essential kubectl logs Commands

The kubectl logs command is the standard tool for retrieving container logs from your terminal.

View Pod Logs

The kubectl logs command is the primary way to view logs from a pod in Kubernetes. It retrieves logs from the container’s stdout and stderr streams, allowing operators and developers to inspect application behavior from the command line. The simplest usage is:

kubectl logs <pod-name>

This command displays the logs for the default container in the specified pod. If the pod contains only one container, Kubernetes automatically selects it. Viewing pod logs is useful for troubleshooting application errors, startup failures, or unexpected behavior without direct access to the node hosting the container.

Follow/Stream Logs

Kubernetes supports real-time log streaming using the -f or --follow flag. This behaves similarly to the Linux tail -f command and continuously outputs new log entries as they are generated by the container.

kubectl logs -f <pod-name>

Streaming logs is useful during debugging sessions or while monitoring deployments and application startup processes. It allows operators to observe live events, detect failures, and verify that applications are functioning correctly after configuration or code changes.

Logs for a Specific Container

When a pod contains multiple containers, Kubernetes requires the container name to identify which logs to display. This is common in pods using sidecar containers for logging, proxies, or monitoring agents.

kubectl logs <pod-name> -c <container-name>

The -c flag specifies the target container within the pod. Without this option, Kubernetes may return an error if multiple containers exist. This command helps isolate logs from a particular component in multi-container pod architectures.

Logs From a Crashed/Previous Pod

Kubernetes can retrieve logs from a previously terminated container using the --previous flag. This is useful when containers crash and restart before administrators can inspect the original logs.

kubectl logs --previous <pod-name>

For pods with multiple containers, the container name can also be specified with -c. Accessing previous logs helps diagnose crash loops, startup failures, and transient application errors that may not appear in the current container instance.

Limit Output

Large log files can be difficult to analyze, so Kubernetes provides options to limit the amount of output returned. The --tail flag displays only the most recent log lines.

kubectl logs --tail=100 <pod-name>

This command returns the last 100 lines of logs, helping users focus on recent events without scrolling through excessive output.

Filter by Time

Kubernetes allows filtering logs based on time using the --since and --since-time flags. These options help narrow log output to a relevant time window during incident investigations.

kubectl logs --since=1h <pod-name>

The example above returns logs generated within the last hour. Kubernetes also supports exact timestamps:

kubectl logs --since-time=2026-05-28T10:00:00Z <pod-name>

Time-based filtering is useful for correlating logs with deployments, outages, or alerts.

Common Kubernetes Logging Use Cases 

Here are some of the most common Kubernetes logging use cases:

• Debugging crashing pods: Loggins is important for diagnosing pods that repeatedly crash or enter a CrashLoopBackOff state. Application logs often reveal the root cause, such as configuration errors, missing dependencies, failed database connections, or runtime exceptions. Using commands like kubectl logs --previous, operators can inspect logs from terminated containers before they restart.
• Investigating failed deployments: Deployment failures can occur due to invalid container images, readiness probe failures, resource constraints, or configuration issues. Kubernetes logs help teams understand why newly deployed workloads fail to start or become healthy. Application logs, combined with Kubernetes events and controller logs, provide visibility into the deployment lifecycle.
• Monitoring application errors: Application logs are used for monitoring runtime errors and identifying performance issues in production environments. Logging systems can aggregate logs from all application instances and detect patterns such as repeated exceptions, HTTP 500 responses, or timeout errors. This allows teams to identify incidents before they affect users.
• Auditing user and system activity: Kubernetes audit logs provide a detailed record of actions performed against the API server. Organizations use these logs to track user activity, monitor administrative changes, and investigate security incidents. Audit logs capture information such as the authenticated user, request type, accessed resource, and response status.
• Troubleshooting node and runtime issues: Node and container runtime logs help diagnose infrastructure-level problems that affect Kubernetes workloads. Issues such as kubelet failures, networking problems, disk pressure, or container runtime crashes are typically identified through system component logs stored on cluster nodes.

Kubernetes Logging Best Practices 

Here are some of the ways that organizations can improve their logging strategy in Kubernetes environments.

1. Centralize Logs Across Clusters, Namespaces, and Workloads

Centralized logging is a best practice for Kubernetes environments, especially in organizations operating multiple clusters or teams. Aggregating logs into a single platform allows operators to search, analyze, and correlate events across namespaces, workloads, and infrastructure components. Without centralized logging, troubleshooting becomes difficult because logs remain fragmented across nodes and clusters.

Organizations commonly use tools such as Fluent Bit, Vector, or Logstash to collect logs and forward them to centralized backends like Elasticsearch, Loki, Splunk, or cloud-native logging platforms. A centralized approach improves operational visibility, simplifies incident response, and ensures logs remain accessible even when workloads are rescheduled or nodes are terminated.

2. Correlate Logs with Kubernetes Resource Metrics

Logs become more valuable when combined with metrics from Kubernetes resources such as pods, nodes, deployments, and containers. Correlating logs with CPU usage, memory consumption, restart counts, or network metrics helps teams identify the root cause of performance issues and failures more quickly.

For example, a spike in application errors may correspond with memory exhaustion or increased latency on a node. Integrating logging systems with observability platforms such as Prometheus and Grafana enables operators to analyze logs and metrics together, providing a more complete understanding of cluster behavior and application health.

3. Use Structured Logging with Kubernetes Metadata

Structured logging improves log searchability and automation by formatting logs as machine-readable data, typically JSON. Instead of relying on unstructured text, structured logs expose fields such as timestamps, severity levels, request IDs, and service names in a consistent format.

Including Kubernetes metadata enhances observability. Log collectors can enrich logs with details such as pod name, namespace, node name, container image, and labels. This metadata allows teams to filter logs by workload, environment, or application owner, making troubleshooting and analysis more efficient in large Kubernetes deployments.

4. Preserve Logs for Restarted, Evicted, and Deleted Pods

Containers and pods in Kubernetes are ephemeral, meaning logs can disappear when workloads restart, crash, or are deleted. To prevent data loss, logs should be exported to durable external storage as soon as they are generated.

Organizations typically deploy log collectors as DaemonSets to ship logs from nodes to centralized storage systems. Retaining logs outside the cluster ensures that historical data remains available for troubleshooting, auditing, compliance, and post-incident analysis, even after the original workload no longer exists.

5. Control Log Volume to Avoid Observability Waste

Excessive logging can increase storage costs, consume network bandwidth, and reduce the efficiency of observability platforms. High-volume Kubernetes environments can generate large amounts of log data, especially when applications emit verbose debug-level logs in production.

Teams should implement logging policies that define appropriate log levels, retention periods, and filtering rules. Reducing unnecessary logs, sampling repetitive events, and excluding low-value data helps control operational costs while maintaining useful observability data.

6. Align Logs with Workload Ownership and FinOps Labels

Applying consistent labels and metadata to workloads improves accountability and cost visibility in Kubernetes environments. Logs enriched with ownership information, team identifiers, environments, and cost allocation labels allow organizations to track which applications generate the most logging data.

This practice supports FinOps initiatives by helping teams understand observability spending and optimize resource usage. It also simplifies operational workflows by allowing logs to be filtered according to business units, services, or deployment environments. Consistent labeling strategies improve governance, troubleshooting efficiency, and cost management across Kubernetes platforms.

Move Beyond Logs to Proactive Kubernetes Troubleshooting with PerfectScale

Logs tell you what happened inside your cluster, but turning that raw output into fast root cause analysis across ephemeral pods, nodes, and namespaces is where teams get stuck. PerfectScale by DoiT delivers AI-guided Kubernetes visibility and governance that combines observability, monitoring, and continuous analysis of performance, waste, and resource metrics across clusters, namespaces, workloads, and node groups. By analyzing telemetry such as logs, traces, and metrics, it helps teams understand not just what is failing but why, so they can uncover unknowns and run root cause analysis with far less manual digging.

Key capabilities of PerfectScale:

• Full-stack visibility and observability: Continuously analyzes real-time and historical metrics across clusters, namespaces, workloads, and node groups, with comprehensive visualization, event tracking, and in-depth insights that keep environment health under control.
• Root cause analysis from telemetry: Goes beyond predefined dashboards by analyzing telemetry data such as logs, traces, and metrics to explain why an issue is happening, helping teams uncover unknown problems and proceed with RCA.
• Impact-aware alerting: Focuses your team on the most important and impactful areas with automatic issue prioritization and budget guardrails, cutting through alerting noise.
• Anomaly detection: Keeps teams instantly informed with cost and resiliency anomaly detection to prevent disruptions and unexpected bills before they reach users.
• Proactive issue remediation: Maintains 99.99 system uptime with AI-driven, predictive intelligence that detects and addresses risks like CPU throttling, OOM, misconfigured requests and limits, and inefficient scaling before they cause pod restarts, performance degradation, or downtime.
• Integration with your existing stack: Connects your established collaboration, ticketing, and observability tools so optimization and troubleshooting fit seamlessly into the workflows your teams already use.

Ready to turn log data into proactive, resilient operations? Explore PerfectScale's AI-guided Kubernetes visibility and governance.