Kubernetes DaemonSet: Tutorial, Use Cases, and Best Practices

Kubernetes DaemonSet Tutorial

This tutorial explains how to create and work with a DaemonSet. It is adapted from the Kubernetes documentation.

1. Create a Kubernetes DaemonSet

To create a Kubernetes DaemonSet, define it in a YAML manifest using apiVersion: apps/v1 and kind: DaemonSet. The manifest must include standard Kubernetes fields such as metadata and spec, along with a pod template under .spec.template. The DaemonSet’s .spec.selector must match the labels assigned to the pod template; after the DaemonSet is created, this selector cannot be changed. Kubernetes also requires the pod’s restart policy to be Always, either explicitly or through the default setting.

For example, the following manifest creates a simple DaemonSet that runs a container on each eligible node:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: example-daemonset
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: example
  template:
    metadata:
      labels:
        app.kubernetes.io/name: example
    spec:
      containers:
      - name: pause
        image: registry.k8s.io/pause

Administrators can extend the pod template with options such as resource requests, volume mounts, tolerations, node selectors, and node affinity. These settings determine how the DaemonSet pods run and, when scheduling constraints are specified, which nodes are eligible to host them. The Kubernetes documentation provides a basic example in which the DaemonSet schedules one pod on every applicable node.

2. Deploy a DaemonSet

Once the DaemonSet manifest has been created, deploy it to the cluster with kubectl apply. If the manifest is saved locally as daemonset.yaml, run:

kubectl apply -f daemonset.yaml

Kubernetes also demonstrates applying a DaemonSet directly from a hosted YAML manifest. For example:

kubectl apply -f https://k8s.io/examples/application/basic-daemonset.yaml

The kubectl apply command submits the configuration to the Kubernetes API, after which the DaemonSet controller determines which nodes are eligible and creates the required pods. As additional eligible nodes join the cluster, the controller automatically creates DaemonSet pods for them. Pods associated with nodes that are removed are subsequently garbage collected.

After deployment, changes to the DaemonSet can generally be managed by modifying the manifest and running kubectl apply again. For DaemonSets using the RollingUpdate strategy, changes to the pod template trigger a rolling update of the managed pods. Kubernetes also supports an OnDelete update strategy when administrators want replacement pods to be created only after existing ones are manually deleted.

3. Check DaemonSet Pods

After deploying a DaemonSet, verify that its pods are running on the expected nodes. The Kubernetes documentation recommends listing pods with the -o wide option, which includes the node assigned to each pod:

kubectl get pods -o wide

For a DaemonSet intended to run throughout the cluster, the output should show a DaemonSet pod on each eligible node. The NODE column makes it easy to confirm that the pods are distributed as expected.

In environments containing many workloads, use the labels defined in the DaemonSet pod template to limit the results. For the example above, the command would be:

kubectl get pods -l app.kubernetes.io/name=example -o wide

You can also inspect the DaemonSet resource itself with:

kubectl get daemonset example-daemonset

or use the abbreviated resource name:

kubectl get ds example-daemonset

The DaemonSet status helps administrators compare the desired number of pods with the number that are current, ready, and available. If individual pods are not running correctly, commands such as kubectl describe pod <pod-name> and kubectl logs <pod-name> can then be used to investigate scheduling or container-level problems. Kubernetes’ basic DaemonSet tutorial verifies successful deployment by checking that the resulting pod list contains one pod for every applicable node.

Kubernetes DaemonSet Best Practices

1. Right-size CPU and Memory Requests and Limits

Setting appropriate CPU and memory requests and limits is critical for DaemonSet stability. Requests define the minimum resources a pod requires, while limits set the maximum it can consume. Over-provisioning wastes resources and may cause scheduling failures on resource-constrained nodes. Under-provisioning risks pod eviction or performance degradation. Analyze the workload’s resource profile to determine reasonable defaults, ensuring that each DaemonSet pod fits comfortably on every targeted node.

Regularly review and adjust resource settings as the application evolves. Resource requirements may change due to new features, increased load, or changes in underlying infrastructure. In dynamic Kubernetes environments, periodic tuning of requests and limits prevents resource contention and supports efficient cluster utilization. Integrate resource reviews into your change management process for ongoing optimization.

2. Base Resource Settings on Historical Usage

Resource requests should reflect how DaemonSet pods behave under real workloads rather than relying only on initial estimates. Use historical CPU and memory metrics from tools such as Prometheus, Grafana, or your cloud monitoring platform to identify typical usage, peak demand, and differences between nodes. Set requests high enough to support normal operation while avoiding unnecessary resource reservation across every targeted node.

Review percentile-based metrics over representative periods instead of using short-term averages. CPU requests can often be based on sustained usage with room for bursts, while memory requests should account for observed peaks. Because DaemonSets multiply their resource footprint across nodes, even small improvements to per-pod settings can free significant cluster capacity.

3. Treat Memory More Conservatively Than CPU

CPU and memory limits have different failure modes in Kubernetes. When a container reaches its CPU limit, Kubernetes throttles its CPU usage, which usually causes slower processing. When a container exceeds its memory limit, however, it can be terminated by the kernel with an out-of-memory (OOM) error. For node-level agents, repeated restarts can create gaps in logging, monitoring, or security coverage.

Set memory requests and limits with enough headroom for temporary spikes and workload growth. Monitor peak memory usage and OOM events before reducing allocations. For agents with unpredictable memory requirements, a restrictive memory limit can be more disruptive than moderate over-allocation, so validate limits against production usage patterns.

4. Use the Appropriate Update Strategy

DaemonSets support RollingUpdate and OnDelete update strategies. RollingUpdate, which is the default, automatically replaces existing pods when the DaemonSet pod template changes. Settings such as maxUnavailable and maxSurge can control how many DaemonSet pods are unavailable or temporarily added during an update.

Use OnDelete when pod replacement must be explicitly controlled. With this strategy, updating the pod template does not automatically replace existing pods; administrators must delete them before Kubernetes creates pods from the new template. Whichever strategy you choose, consider the effect of unavailable agents on logging, monitoring, networking, or security coverage during updates.

5. Continuously Monitor Efficiency as the Cluster Changes

DaemonSet resource consumption grows with the number of eligible nodes. A pod that uses modest resources on one node can represent substantial aggregate CPU and memory consumption in a large cluster. Track both per-pod usage and the total resources consumed by each DaemonSet to understand its effect on cluster capacity.

Reassess DaemonSets when node types, workloads, or cluster size change. Watch for unused resource requests, CPU throttling, memory pressure, OOM events, restarts, and pods that fail to schedule. Continuous monitoring helps identify when resource settings or scheduling rules no longer match actual operating conditions.

Optimizing DaemonSet Resource Usage with PerfectScale

Because a DaemonSet places a pod on every eligible node, even small amounts of over- or under-provisioning multiply across the cluster. PerfectScale by DoiT is a Kubernetes optimization platform that deploys via a single Helm command and then delivers actionable insights and autonomous optimization across the entire K8s stack, so node-level agents such as log collectors, monitoring exporters, and security daemons run with the resources they actually need.

Key capabilities of PerfectScale:

• Autonomous workload right-sizing: Podfit gives a granular view of cluster health and costs, prioritizes the areas that need attention, and autonomously optimizes workloads with data-driven right-sizing recommendations that can be applied automatically.
• Node utilization optimization: Infrafit provides comprehensive node utilization visibility and granular insights that identify and eliminate idle node capacity, and helps select the right nodes for your workloads.
• Autoscaler alignment: Works alongside HPA, KEDA, Karpenter, Cluster Autoscaler, EKS Auto Mode, Fargate, Node Auto Provisioning, and Google Autopilot, with actionable recommendations to improve their configurations.
• Real-time alerting with prioritization: Advanced auto-prioritization surfaces the most impactful resilience risks, cost spikes, and anomalies before they affect users or the cloud bill, with alerts delivered to Slack, Datadog, MS Teams, or PagerDuty.
• Trends and governance reporting: The Trends report provides in-depth visibility into cost, waste, and risk metrics over time across clusters, node groups, namespaces, and workloads, supporting forecasting and root cause analysis.
• Broad environment support: Runs on on-premise and cloud-based clusters including OpenShift, EKS, GKE, and AKS, and supports Windows-based containers as well as ephemeral and ML workloads.

Learn more about the PerfectScale platform