How to Fix Kubernetes Resource Policies That Optimize for the Wrong Phase

TL;DR

• A Kubernetes resource policy is every layer that decides what requests and limits a container ends up with: ResourceQuota, LimitRange, admission rules (Kyverno, Gatekeeper, ValidatingAdmissionPolicy), Helm chart defaults, and whatever recommender writes the numbers.
• Every one of those layers assumes one number per container for the container's whole life. Containers have two phases, startup and steady state, and the gap between them can be 3x to 10x.
• Policies tuned for steady state throttle and OOM-kill pods during startup. Policies tuned for startup reserve capacity nobody uses, fill namespace quotas with idle requests, and leave pods Pending while the cluster sits half empty.
• The fix is structural: quota on requests, not limits; LimitRange that allows bursting instead of clamping it; admission rules that require requests and memory limits but stop mandating CPU limits; and a recommender that sees startup and steady state as separate numbers.
• In-place pod resize makes the last step practical. Startup gets what it needs, then the pod shrinks to its plateau without a restart, and quota is reclaimed at the same moment.

In the last post we covered how to see a startup spike: filter throttling and OOM metrics by container age, and the burst that averages hide shows up in the first 60 seconds of every container's life. This post is about what to do once you can see it, because the hard part isn't editing one Deployment. It's that the policy layer around that Deployment was written for one phase and will keep pushing every workload back toward it.

Fix 4: Size memory for startup, CPU for steady state

Once the policy layer stops forcing a single answer, the per-workload decision gets simpler, and it's different for each resource.

Memory limit clears the startup peak. There is no graceful degradation for memory. If initialization allocates 900Mi and the limit is 768Mi, the container dies with exit code 137 before it ever reaches its 400Mi plateau. Take max_over_time(container_memory_working_set_bytes[5m]) over a window that includes a rollout and set the limit above that number with margin. For JVM workloads remember that heap is only part of the picture: metaspace, code cache, and thread stacks all sit outside -Xmx.

Memory request sits at the plateau. The request is what the scheduler reserves and what quota counts. Setting it to the steady-state working set with modest headroom keeps bin-packing honest. The gap between request and limit puts the pod in Burstable QoS, which means it's a candidate for eviction under node memory pressure before Guaranteed pods are. For most stateless services that is an acceptable trade; for the ones where it isn't, set request equal to limit deliberately and accept the reservation.

CPU request sits at the plateau, CPU limit is absent. With no CFS quota, startup borrows idle cycles from the node and gives them back a minute later. The request still matters: it's what Karpenter and the scheduler use to place the pod, and it's what HPA divides utilization by. Size it to steady-state p95, not the burst.

The one place this breaks is a rollout that lands many spiky pods on the same fresh node at once. With no idle CPU to borrow, startup runs slow through contention rather than throttling. Bound maxSurge so new pods arrive in waves rather than all at once, and consider a topologySpreadConstraint on kubernetes.io/hostname so a rollout's new pods don't stack on one node. A startupProbe with a failureThreshold sized to the slow case keeps the kubelet from restarting a pod that's merely booting under load.

Fix 5: Let the recommender see two phases, and let the pod change size

Everything above stops the policy from forcing the wrong number. It doesn't produce the right one. For that the recommender has to know that startup and steady state are different, and the pod has to be able to move between them.

The Vertical Pod Autoscaler on its default recommender builds a decaying histogram over roughly eight days of samples. A 45-second burst is noise to it, so it recommends the plateau, and the next rollout throttles. You can constrain it with minAllowed so it never drops below a floor that survives boot, but that just re-encodes the startup number as a permanent reservation through a different door.

In-place pod resize changes the shape of the problem. It went stable in Kubernetes 1.35 and is on by default in 1.36, and it lets a running container's requests and limits change without a restart. The pattern is: start the pod with enough CPU and memory to boot cleanly, then resize it down to steady-state values once it reports ready. GKE's startup CPU boost does a version of this for Autopilot. The pod spec just needs to declare that a CPU change shouldn't trigger a restart:

containers:
- name: app
  resizePolicy:
  - resourceName: cpu
    restartPolicy: NotRequired
  - resourceName: memory
    restartPolicy: NotRequired   # or RestartContainer if the runtime can't shrink heap live
  resources:
    requests:
      cpu: "2"          # startup value; resized to ~200m after readiness
      memory: 1Gi
    limits:
      memory: 1Gi

This is where the quota work from Fix 1 pays off. A resize is checked against ResourceQuota like any other request change, so when the pod shrinks from 2 cores to 200m the namespace gets 1.8 cores of quota back immediately. Forty replicas boot in waves, each one holding its startup reservation for about a minute, and the namespace's steady-state footprint settles at 8 cores instead of 80. Pods stop sitting Pending behind reservations that describe a phase nobody is in.

One constraint to design around: a resize can't change a pod's QoS class. A pod that started Burstable stays Burstable, and a Guaranteed pod has to keep requests equal to limits through every resize. That's another reason not to mandate Guaranteed by policy for workloads that don't need it.

A policy rewrite checklist

1. Inventory the layers. For each namespace: kubectl get resourcequota,limitrange -o yaml, plus every Kyverno/Gatekeeper/VAP rule that touches resources. Write down what each one forces.
2. Find the phase each layer assumes. Any limits.cpu in a quota, any max.cpu or low maxLimitRequestRatio.cpu in a LimitRange, any "must set CPU limit" admission rule is a steady-state policy. Any defaultRequest above a few hundred millicores, or any requests-equal-limits mandate, is a startup policy.
3. Measure the gap. Run the requests-to-usage ratio query per namespace. Anything above 3 to 4 is quota consumed by the wrong phase.
4. Move quota to requests. Drop limits.cpu from ResourceQuota; keep requests.* and limits.memory.
5. Open the LimitRange. Remove default.cpu, raise or remove max.cpu, apply maxLimitRequestRatio to memory only.
6. Rewrite admission. Require CPU and memory requests and a memory limit. Stop requiring a CPU limit.
7. Re-size the top ten restarters. Memory limit to startup peak, memory and CPU requests to steady-state p95, no CPU limit. Add maxSurge and spread constraints if rollouts contend.
8. Enable two-phase sizing where the cluster supports it. On 1.33+ with in-place resize available, declare resizePolicy and let a controller shrink pods after readiness.
9. Re-run step 3 a week later. The ratio should fall toward 1.5 to 2 and the young-container throttling rate should fall toward zero at the same time. If only one moved, you fixed one resource and not the other.

FAQ

What is a Kubernetes resource quota? A ResourceQuota is a namespace-scoped object that caps the total resources all pods in that namespace can request or be limited to, along with object counts like pods, services, and PVCs. It's enforced at admission: a pod whose requests or limits would push the namespace over its hard values is rejected. Quotas count the numbers in the pod spec, not actual usage, which is why a quota can be exhausted while the nodes behind it sit idle.

Should a ResourceQuota cap limits or requests? Cap requests.cpu and requests.memory, because requests are what the scheduler reserves and what you pay for in node capacity. Cap limits.memory too, since it's a real safety bound against a namespace consuming all node memory. Avoid limits.cpu in quotas: it pressures teams to shrink CPU limits to fit, which throttles pods during startup without protecting the node.

What's the difference between ResourceQuota and LimitRange? ResourceQuota sets a namespace-wide total. LimitRange sets per-container (or per-pod, or per-PVC) defaults, minimums, maximums, and the allowed ratio between limit and request. Quota answers "how much can this team use in total"; LimitRange answers "what does one container get if it doesn't say, and what's the biggest it can ask for." Both are enforced at admission.

Do in-place pod resizes count against ResourceQuota? Yes. A resize that raises requests is validated against the namespace quota and rejected if it would exceed it. A resize that lowers requests returns the difference to the quota immediately. That's what lets a startup-then-shrink pattern keep a namespace's steady-state quota footprint near its actual usage.

Why do my pods get throttled at startup even though the node has idle CPU? Because a CPU limit is enforced by the kernel's CFS quota per container, independent of node load. If the limit is 500m, the container gets 50ms of every 100ms period regardless of how idle the node is. Startup work like JIT compilation or module loading exhausts that quota early in each period and waits. Removing the CPU limit, or raising maxLimitRequestRatio so the limit can sit well above the request, lets startup use the idle capacity.

Policies that match how containers actually behave

Everything above can be done by hand: rewrite the quota, open the LimitRange, update the admission policy, re-size the noisiest workloads, repeat when the code changes. The hard part is the last clause. Startup profiles shift with every dependency bump and every JVM flag change, and a policy you tuned correctly in March is wrong again by June.

PerfectScale by DoiT profiles every container at the workload level, so startup and steady-state behavior show up as separate patterns rather than one blended average, and its recommendations size memory to clear the boot peak while keeping requests at the plateau. For Java workloads it detects JVM containers automatically once the Coroot agent is enabled, tracks heap, non-heap, and GC over time, flags containers running without explicit heap settings, and respects -Xms and -Xmx when it recommends or applies a change. On clusters running Kubernetes 1.33 or later, its automation applies right-sizing in place without restarting the pod, and falls back to a rolling restart only when a resize isn't feasible on the current node, so the namespace quota reflects what the workloads are using now rather than what they needed for 45 seconds after the last deploy.

Want to see a pod boot at 2 cores and settle at 200m with no restart and no quota left stranded? Join the JVM startup spike workshop, or book a technical session and we'll look at the quota and LimitRange policies in your own clusters.