Kubernetes Tolerations: Examples, Use Cases, and Best Practices

Kubernetes Taint Effects

Let’s review the taint effects offered by Kubernetes, which tolerations can override.

NoSchedule

The NoSchedule effect prevents new pods from being scheduled on a node unless they have a matching toleration. Pods that are already running on the node when the taint is added are not evicted. This makes NoSchedule useful for reserving nodes for specific workloads without disrupting existing pods.

For example, adding a hardware=gpu:NoSchedule taint prevents pods without a matching toleration from being newly scheduled on the GPU node. Pods already running there can continue to run.

PreferNoSchedule

The PreferNoSchedule effect is a soft scheduling restriction. Kubernetes tries to avoid placing pods without a matching toleration on the tainted node, but it can still schedule them there when necessary. Unlike NoSchedule, it does not strictly block scheduling.

This effect is useful when administrators prefer to reserve nodes for certain workloads but still want the scheduler to use those nodes when other placement options are limited. Existing pods are not evicted when a PreferNoSchedule taint is added.

NoExecute

The NoExecute effect affects both scheduling and pods already running on a node. New pods without a matching toleration cannot be scheduled there, while existing pods that do not tolerate the taint are evicted.

A toleration can include tolerationSeconds to let a pod remain on the node temporarily after a matching NoExecute taint appears. After that period expires, the pod is evicted if the taint remains. If tolerationSeconds is omitted, a matching toleration allows the pod to remain while the taint is present.

Kubernetes Tolerations Examples

Example 1: Basic NoSchedule Toleration

Suppose a node has the taint workload=analytics:NoSchedule. A pod needs a matching toleration to become eligible for scheduling on that node. The following pod tolerates this specific taint:

apiVersion: v1
kind: Pod
metadata:
  name: analytics-pod
spec:
  containers:
    - name: web
      image: nginx:latest
  tolerations:
    - key: "workload"
      operator: "Equal"
      value: "analytics"
      effect: "NoSchedule"

The Equal operator requires both the key and value to match the taint. This toleration allows the pod onto nodes with the specified taint, but it does not require the scheduler to place the pod on one of those nodes.

Example 2: Tolerating Any Value for a Taint

The Exists operator can be used when a pod should tolerate a taint key regardless of its value. For example, the following configuration tolerates any NoSchedule taint whose key is workload:

apiVersion: v1
kind: Pod
metadata:
  name: compute-pod
spec:
  containers:
    - name: worker
      image: busybox:latest
  tolerations:
    - key: "workload"
      operator: "Exists"
      effect: "NoSchedule"

No value is specified when using Exists. As a result, this pod can tolerate taints such as workload=database:NoSchedule and workload=batch:NoSchedule. Other taint keys or effects still require separate matching tolerations.

Example 3: NoExecute With tolerationSeconds

A NoExecute toleration can include tolerationSeconds to control how long a pod remains on a node after a matching taint is applied. The following pod tolerates a maintenance-window=active:NoExecute taint for 100 seconds:

apiVersion: v1
kind: Pod
metadata:
  name: maintenance-worker
spec:
  containers:
    - name: worker
      image: busybox:latest
  tolerations:
    - key: "maintenance-window"
      operator: "Equal"
      value: "active"
      effect: "NoExecute"
      tolerationSeconds: 100

If the taint is added while the pod is running, the pod can remain on the node for up to 100 seconds. If the taint still exists after that period, Kubernetes evicts the pod. If the taint is removed before the period expires, the pod can continue running.

Kubernetes Tolerations Best Practices

Use Tolerations Only for Workloads That Need Them

Add tolerations only when a workload has a clear reason to run on tainted nodes. Broad or unnecessary tolerations weaken the isolation that taints are intended to provide and can allow workloads onto nodes reserved for other purposes. This can lead to resource contention and make node placement less predictable.

Review tolerations as workload requirements change. Avoid generic Exists tolerations unless the pod genuinely needs to tolerate a wide range of taints. Prefer narrowly scoped keys, values, and effects so each workload receives only the scheduling permissions it requires.

It is also useful to manage tolerations at the workload-controller level, such as in a Deployment, StatefulSet, or DaemonSet. This keeps scheduling behavior consistent when pods are recreated or scaled.

Pair Tolerations With Node Affinity

A toleration makes a pod eligible to run on a tainted node, but it does not direct the pod to that node. If a workload should specifically run on a particular node pool, combine tolerations with node affinity or node selectors.

For example, a GPU workload can tolerate a GPU-node taint while using node affinity to require nodes labeled with the appropriate GPU type. The taint keeps ordinary workloads away, while affinity directs the GPU workload toward compatible nodes.

Choose between required and preferred node affinity based on how strict placement needs to be. Required affinity prevents scheduling on nodes that do not match, while preferred affinity gives the scheduler more flexibility when suitable nodes are unavailable.

Keep Taints and Tolerations Consistent Across Node Pools

Use a consistent naming scheme for taint keys and values across node pools. Inconsistent values such as workload=gpu, type=gpu, and node=gpu for the same purpose make pod configurations harder to maintain and increase the risk of scheduling errors.

Define standard taints for common node roles and apply them through cluster or infrastructure automation. Workload manifests can then use predictable tolerations across development, staging, and production without unnecessary configuration differences.

Consistency is especially important when nodes are automatically created or replaced by cluster autoscaling systems. New nodes should receive the expected taints and labels during provisioning so workloads behave the same regardless of which node instance is running.

Protect Specialized and Expensive Nodes

Use taints to prevent general-purpose pods from consuming specialized resources such as GPUs, high-memory instances, or other expensive hardware. Only workloads designed to use those resources should receive the corresponding tolerations.

Tolerations alone do not ensure that a pod actually requests the specialized resource. For GPU workloads, for example, configure the appropriate resource requests or limits as well as the toleration. Node affinity can provide additional placement control when several hardware types are available.

This approach helps prevent low-priority workloads from occupying capacity needed by specialized applications. It can also reduce infrastructure costs by keeping expensive nodes available for workloads that can benefit from their hardware.

Monitor Scheduling Outcomes, Not Just Configuration

A valid toleration does not guarantee that a pod will be scheduled successfully. Resource availability, node affinity, topology constraints, pod affinity and anti-affinity, and other scheduler rules can still prevent placement.

Monitor pending pods, scheduler events, node taints, and actual pod placement to verify that scheduling policies behave as intended. Commands such as kubectl describe pod and kubectl describe node can help identify taint mismatches and other scheduling constraints.

Monitoring should also detect unexpected placement, not only pods that remain pending. A workload running successfully on the wrong node pool can indicate overly broad tolerations or missing affinity rules. Periodic checks help ensure scheduling policies continue to work as the cluster and its workloads change.

FAQ

What is a Kubernetes toleration? A toleration is a setting in a pod's spec that lets the pod be scheduled on nodes with matching taints. It only allows scheduling. It does not guarantee the pod lands on a tainted node, because resource requests and node affinity still apply.

What is the difference between a taint and a toleration? Taints are applied to nodes and repel pods that do not have a matching toleration. Tolerations are applied to pods and permit, but do not require, scheduling onto nodes with matching taints.

What is the difference between the Equal and Exists operators? Equal requires the toleration's key, value, and effect to match the taint, and it is the default when no operator is set. Exists matches on the key alone, so the value must be left out, and it tolerates any value for that key.

What do the NoSchedule, PreferNoSchedule, and NoExecute effects do? NoSchedule blocks new pods without a matching toleration, but leaves running pods alone. PreferNoSchedule is a soft version where the scheduler tries to avoid the node. NoExecute also evicts running pods that do not tolerate the taint.

What does tolerationSeconds do? On a NoExecute toleration, tolerationSeconds sets how long a pod can stay on a node after a matching taint appears. Once that time passes and the taint is still there, Kubernetes evicts the pod. If it is left out, the pod stays while the taint is present.

Does a toleration make a pod run on a tainted node? No. A toleration only makes the pod eligible. To direct a pod to a specific node pool, pair the toleration with node affinity or a node selector.

Making Toleration-Based Scheduling More Efficient with PerfectScale

Taints and tolerations decide where pods are allowed to run, but they say nothing about whether those nodes are sized correctly or whether the workloads landing on them are requesting the right amount of CPU and memory. PerfectScale is a Kubernetes optimization and governance platform that deploys via a single Helm command and then delivers actionable insights and autonomous optimization across the full K8s stack, workloads, nodes, and autoscalers, so that reserved, specialized, and expensive node pools are actually used efficiently.

Key capabilities of PerfectScale:

• Autonomous workload right-sizing: Podfit gives a granular view of cluster health and cost, highlights wasted resources and resilience issues, and autonomously optimizes workloads with data-driven right-sizing recommendations you can put on automation.
• Node-level utilization insights: Infrafit surfaces idle node capacity and recommends the right node types for your workloads, so dedicated, GPU, and other tainted node pools deliver peak performance without paying for unused capacity.
• Autoscaler optimization: PerfectScale integrates with HPA, KEDA, Karpenter, Cluster Autoscaler, EKS Auto Mode, Fargate, Node Auto Provisioning, and Google Autopilot, maximizing the effectiveness of the systems that provision and replace your tainted nodes.
• Real-time alerts with auto-prioritization: Resilience risks and cost anomalies are surfaced with impact-driven prioritization and delivered straight to Slack, Datadog, MS Teams, or PagerDuty before they reach your users or your cloud bill.
• Trends, governance, and forecasting: The Trends report provides in-depth visibility into cost, waste, and risk metrics over time across clusters, node groups, namespaces, and workloads, supporting root cause analysis and accurate budget planning.
• Any Kubernetes environment: PerfectScale runs on both on-premise and cloud-based clusters, including OpenShift, EKS, GKE, AKS, and hybrid setups, with support for Windows containers and ephemeral or ML workloads such as Airflow and Spark jobs.

Learn more about the PerfectScale platform