Kubernetes v1.37 (Garhwal): What's New and Improved

TL;DR

Kubernetes v1.37 ("Garhwal") ships 67 enhancements: 16 Stable, 23 Beta, 27 Alpha, and 1 removal. The headline changes are in-place resize for init and sidecar containers (GA), configurable HPA tolerance (GA), node declared features (GA), scale-to-zero for the HorizontalPodAutoscaler (Beta), Memory QoS (Beta), pod-level resource managers (Beta), gang scheduling via the new Workload API (Beta), and Kubelet rootless mode (Beta). Alpha work includes a cooperative Eviction Request API and pod-level checkpoint/restore, plus continued progress on Dynamic Resource Allocation (DRA). Action item: SELinux mount labeling changes by default in this release and can break existing workloads on SELinux-enabled clusters.

Kubernetes v1.37, also known as "Garhwal," brings a strong set of changes for teams that care about resource management, autoscaling, and scheduling. This release has 67 enhancements. 16 of them are now Stable (GA), 23 are now Beta, 27 are now Alpha, and 1 is a removal. The name comes from Garhwal, a Himalayan region in Uttarakhand, India. The theme is about layers that connect and support each other, from terraced fields to mountain streams, which fits a release that keeps building on work from earlier versions.

These are some of the features we are most excited about in Kubernetes v1.37: in-place resize for init and sidecar containers, scale to zero for the HorizontalPodAutoscaler, Memory QoS, and native gang scheduling through the Workload API. Many of these make Kubernetes better at using resources well, scaling on real demand, and placing tightly coupled workloads.

Here are the major changes in Kubernetes v1.37:

Kubernetes v1.37 Stable Features

1. In-place Pod Resize for Init and Sidecar Containers

Feature Group: SIG Node | KEP: #1287

In-place Pod resize became Stable (GA) in Kubernetes v1.35. In v1.37, this support extends to init containers and sidecar containers, and that part is now GA too.

Before this change, you could resize the CPU and memory of a running app container without a restart, but init and sidecar containers were left out. A sidecar container is an init container with restartPolicy: Always, so it runs for the whole life of the Pod. If a sidecar needed more CPU or memory, your only real option was to recreate the Pod. That meant downtime for the main app as well, even though only the sidecar needed the change.

With v1.37, the InPlacePodVerticalScalingInitContainers feature gate is GA. You can change the CPU and memory requests or limits of a running sidecar container, and the kubelet updates the container's cgroup settings in place. The Pod keeps running. You control what happens per resource with resizePolicy. NotRequired means apply the change live, and RestartContainer means restart only that container to apply it. v1.37 also adds two Alpha pieces to the same feature: resizing memory-backed emptyDir volumes in place (the InPlacePodVerticalScalingMemoryBackedVolumes gate), and letting the scheduler preempt lower-priority Pods to make room for a deferred resize (the InPlacePodVerticalScalingSchedulerPreemption gate).

apiVersion: v1
kind: Pod
metadata:
  name: app-with-sidecar
spec:
  initContainers:
  - name: log-shipper        # sidecar
    image: log-shipper:1.0
    restartPolicy: Always
    resizePolicy:
    - resourceName: cpu
      restartPolicy: NotRequired
    - resourceName: memory
      restartPolicy: RestartContainer
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"
      limits:
        cpu: "200m"
        memory: "256Mi"
  containers:
  - name: app
    image: app:1.0
    # ...

With this setup, you can raise the sidecar's CPU with no restart at all. Raising its memory restarts only that one container, never the whole Pod.

2. Configurable Tolerance for HorizontalPodAutoscalers

Feature Group: SIG Autoscaling | KEP: #4951

In Kubernetes v1.37, configurable tolerance for the HorizontalPodAutoscaler (HPA) has graduated to General Availability (GA).

The HPA changes the number of Pod replicas based on metrics like CPU or memory. Before this feature, the HPA used a fixed 10% tolerance across the whole cluster. It would not act until usage moved more than 10% away from the target. That single value did not fit every workload. Sensitive services could react too slowly, and to change it, you had to edit a global controller flag that touched every HPA in the cluster.

With v1.37, the HPAConfigurableTolerance feature gate is GA. You can set the tolerance per HPA and per scaling direction in the behavior field. A tight tolerance on scale up lets a critical service react to small load increases. A looser tolerance elsewhere keeps quieter workloads stable. This was Beta in v1.35 and is now the standard behavior.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  behavior:
    scaleUp:
      tolerance: 0.05

The tolerance is a ratio, not a number of percentage points. With 0.05 and a 60% target, the HPA scales up once CPU usage passes about 63% (60% times 1.05). That reacts sooner than the old fixed 10% tolerance would allow.

3. Node Declared Features

Feature Group: SIG Node | KEP: #5328

In Kubernetes v1.37, node declared features has graduated to General Availability (GA).

Before this feature, Kubernetes assumed a node could handle any feature the control plane supported, as long as versions were within the supported skew. That is not always true. When a new feature was turned on at the control plane, the scheduler could still place a Pod that used it onto an older node that did not support it yet. The Pod would then fail at runtime or get stuck after scheduling, even though the placement looked fine.

With v1.37, the NodeDeclaredFeatures feature gate is GA. Each node now publishes the set of features it understands in a status.declaredFeatures field on the Node object. The scheduler and admission checks use this to place feature-using Pods only on nodes that support those features. This turns a runtime failure into a clear scheduling decision, which matters most during upgrades when node versions are mixed. Node declared features was Alpha in v1.35.

Kubernetes v1.37 Beta Features

4. Scale to Zero for HorizontalPodAutoscalers

Feature Group: SIG Autoscaling | KEP: #2021

In Kubernetes v1.37, scale to zero for the HorizontalPodAutoscaler has graduated to Beta and is on by default.

Before this feature, an HPA could not take a workload down to zero replicas. The lowest it would go was one. For a service that only has work some of the time, like a queue consumer or a batch worker, that meant paying to keep at least one Pod running even when there was nothing to do.

With v1.37, you can set spec.minReplicas: 0 on an HPA and let it scale the workload down to zero when it is idle, then bring it back when demand returns. This works for workloads driven by Object or External metrics, such as a queue length. It does not work for CPU or memory metrics because those metrics come from running Pods, and there are none at zero. This can cut cost for queue consumers, batch jobs, and GPU workloads that sit idle between runs. While the HPA holds a workload at zero, it records a ScaledToZero condition set to True, so you can tell the difference between a workload it scaled to zero and one that was turned off by hand. Once it scales back up, the condition becomes False with the reason NotScaledToZero.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: queue-worker
  minReplicas: 0
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: queue_messages
      target:
        type: AverageValue
        averageValue: "30"

With this configuration, the workload drops to zero Pods when the queue is empty, and the HPA scales it back up once messages arrive.

5. Memory QoS

Feature Group: SIG Node | KEP: #2570

In Kubernetes v1.37, Memory QoS has graduated to Beta.

Before this feature, the kubelet set a hard memory limit for a container (cgroup memory.max), but it did not give the kernel any softer signals. A container could use memory right up to its limit and then get OOM killed with little warning. There was also no memory protection tied to a container's request, so under memory pressure the kernel could reclaim pages from a container that was still inside its request.

With v1.37, the MemoryQoS feature gate is Beta. It uses cgroup v2 memory controls to give the kernel better signals. The kubelet sets memory.min from a container's memory request, which protects that much memory from reclaim. When you set memoryThrottlingFactor in the kubelet config, the kubelet also sets memory.high as a throttling point below the limit. The kernel then slows the container down and reclaims pages before it hits the hard limit and gets OOM killed. By default memoryThrottlingFactor is not set, and memory.high is left off unless you configure it. cgroup v2 is required, which is already the only supported mode since v1.35.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

With this setting, the kubelet sets memory.high for containers below their memory limit, using the 0.9 factor. The kernel reclaims memory and throttles the container before it reaches the hard limit, which makes OOM kills less likely under pressure.

6. Pod-level Resource Managers

Feature Group: SIG Node | KEP: #5526

In Kubernetes v1.37, pod-level resource managers has graduated to Beta.

Kubernetes v1.34 added pod-level resources (KEP #2837), which let you set CPU and memory requests and limits for the whole Pod instead of only per container. But the kubelet's resource managers did not understand those pod-level values yet. Before this feature, the CPU Manager, Memory Manager, and Topology Manager only looked at per-container requests. A Pod that set resources at the Pod level could miss out on exclusive CPU pinning or NUMA alignment.

With v1.37, the PodLevelResourceManagers feature gate is Beta. It is off by default in v1.37: it was turned on, then switched back off late in the release cycle after the team found critical issues, so you enable it yourself if you want it. When it is on, the CPU Manager (static policy), Memory Manager, and Topology Manager account for pod-level resources when they assign CPUs, memory, and hardware locality. This helps latency-sensitive and high-end workloads that set resources at the Pod level, so they still get exclusive cores and NUMA-aligned memory.

apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-pod
spec:
  resources:              # pod-level resources
    requests:
      cpu: "2"
      memory: "4Gi"
    limits:
      cpu: "2"
      memory: "4Gi"
  containers:
  - name: app
    image: app:1.0
    # containers can set their own resources, or use the pod-level ones

With this configuration, the Pod is Guaranteed at the Pod level, and the kubelet's managers can pin whole CPUs and align memory for it, even when the container does not set its own requests.

7. Gang Scheduling with the Workload API

Feature Group: SIG Scheduling | KEP: #4671

In Kubernetes v1.37, native gang scheduling through the Workload API has graduated to Beta.

Before this feature, Kubernetes scheduled Pods one at a time. For a tightly coupled job, like a distributed training run where every worker must start together, this caused partial scheduling. Some Pods would start while others stayed pending, holding resources and getting nothing done. Teams had to add an external scheduler to get all-or-nothing (gang) behavior. Kubernetes v1.35 introduced native gang scheduling as Alpha through a new Workload API.

With v1.37, the API is now scheduling.k8s.io/v1beta1. You turn it on with the GenericWorkload feature gate. In v1.37, the older GangScheduling and WorkloadAwarePreemption gates were merged into GenericWorkload, so one gate now covers both gang scheduling and workload-aware preemption. The model has three parts. A Workload holds one or more podGroupTemplates, each with a scheduling policy such as gang and a minCount. A PodGroup is created from a template and tracks one running group. Each Pod points at its group with schedulingGroup.podGroupName. The scheduler holds the Pods until at least minCount of them can run together and then places them. If it cannot meet minCount, none are bound, and the Pods wait until there is room. The minCount field stays changeable after creation, which helps elastic jobs scale up or down.

The Workload defines the template:

apiVersion: scheduling.k8s.io/v1beta1
kind: Workload
metadata:
  name: training-job
spec:
  podGroupTemplates:
  - name: workers
    schedulingPolicy:
      gang:
        minCount: 4

A PodGroup is created from that template:

apiVersion: scheduling.k8s.io/v1beta1
kind: PodGroup
metadata:
  name: training-job-workers-0
spec:
  workloadRef:
    workloadName: training-job
    templateName: workers
  schedulingPolicy:
    gang:
      minCount: 4

Each Pod joins the group by name:

apiVersion: v1
kind: Pod
metadata:
  name: worker-0
spec:
  schedulingGroup:
    podGroupName: training-job-workers-0
  containers:
  - name: trainer
    image: trainer:1.0
    # ...

With this setup, Kubernetes schedules the workers only when at least four of them can run together. If the cluster cannot fit four, none of them start.

8. Kubelet in User Namespace (Rootless Mode)

Feature Group: SIG Node | KEP: #2033

In Kubernetes v1.37, running the kubelet in a user namespace, also called Rootless mode, has graduated to Beta.

Before this feature, node components like the kubelet ran as root on the host. If the kubelet was compromised, an attacker could gain root on the whole node.

With v1.37, the KubeletInUserNamespace feature gate is Beta. It runs the kubelet and the other node components inside a Linux user namespace. They still look like root inside the namespace, but on the host they map to an unprivileged user. This adds a layer of isolation without changing how you use the kubelet day to day. This feature has been in the works since v1.22, so reaching Beta is a big step.

Kubernetes v1.37 Alpha Features

9. Eviction Request API

Feature Group: SIG Node | KEP: #4563

Before this feature, draining a node for maintenance was a client-side action. A tool like kubectl drain evicted Pods one by one and respected PodDisruptionBudgets, but the intent lived in the client, not in the cluster. Controllers and applications could not see that a node was being drained, coordinate a safe handoff, or take part in the decision. A stateful app could get evicted at a bad moment because nothing told it maintenance was coming.

With v1.37, an Alpha feature adds the EvictionRequest API so eviction can be cooperative instead of a hard delete. A requester creates an EvictionRequest for a Pod, and components that registered through the new spec.evictionResponders Pod field handle a graceful handoff. v1.37 also adds new Node lifecycle conditions (MaintenancePlanned, MaintenanceInProgress, GracefulNodeShutdownInProgress, DrainInProgress, and Drained) that controllers can watch to see where a node is in the maintenance flow. Together, these let an application or operator drain on its own terms, for example by moving a leader or flushing state before its Pod goes away.

10. Pod-level Checkpoint and Restore

Feature Group: SIG Node | KEP: #5823

Before this feature, Kubernetes could checkpoint a single container (KEP #2008), mainly for forensic analysis. You could snapshot one container's state, but not a whole Pod at once, so a multi-container Pod could not be captured as one unit.

With v1.37, an Alpha feature adds Pod-level checkpoint and restore. It extends the Container Runtime Interface (CRI) with CheckpointPod and RestorePod calls, so the kubelet and a compatible runtime can snapshot every container in a Pod together, then restore them. This is useful for forensic work today, and it lays the groundwork for future uses like moving a running Pod between nodes. The feature needs a container runtime that supports the new CRI calls.