▶️ On-Demand Kubernetes Excellence Workshop
Why Kubernetes Cost Optimization Breaks Production & How to Fix It
Every SRE team running Kubernetes is walking a fine line: optimize for cost, and risk OOM kills, throttling, and outages, or optimize for reliability, and infrastructure costs keep climbing. There's a better way.
Overview
Every SRE team running Kubernetes is walking a fine line: optimize for cost, and you risk OOM kills, throttling, and outages; optimize for reliability, and infrastructure costs keep climbing.
Most cost-optimization tools chase the first half of that equation — and lose sight of the second. In their pursuit of maximum savings, they lure engineers into ripping and replacing existing autoscaling solutions, demand wide write permissions to production clusters, and switch nodes with no business justification.
The result: degraded uptime and painful incidents. So most engineers, once burned, turn automation off entirely — leaving money on the table instead.
There's a better way.
Join Anton Weiss and Arthur Berezin as they break down how reliability-first optimization makes real cost reduction possible — through workload-aware guardrails that enable continuous, autonomous optimization without the production risk.
By watching this session on-demand, you'll walk away understanding:
- Why traditional cost-optimization approaches introduce reliability risk — and where the failure points actually happen
- How workload-aware guardrails enable safe, continuous automation without full rip-and-replace changes to your existing autoscaling stack
- What a reliability-first optimization strategy looks like in practice, and how it unlocks cost savings teams previously left on the table
Ready to Optimize Kubernetes Costs Without the Risk?
See how PerfectScale's workload-aware guardrails cut Kubernetes costs - without risking production.
On-Demand K8s Excellence Workshop
Meet this session's instructors
Click an image to learn more about the presenters
The State of Kubernetes Clusters Today
Drawing on data from hundreds of production clusters analyzed over three years, Anton Weiss and Arthur Berezin opened with the numbers behind the cost-vs-reliability tradeoff:
- Unoptimized production clusters waste an average of $5,000–$10,000 per month
- 80% of unoptimized clusters are over-provisioned
- Even in over-provisioned clusters, 1 in 10 workloads is high-risk (under-provisioned in a way that threatens reliability)
Why Engineers Don't Optimize
Despite the waste, teams avoid optimization for a few consistent reasons:
- Fear of outages: nobody wants to be the reason a service goes down, or get paged at night
- Kubernetes complexity: clusters run dozens to hundreds of workloads, each needing individually tuned CPU/memory requests and limits; manual optimization doesn't scale
- Split incentives: SREs are measured on SLOs and error budgets, not cloud cost; FinOps owns cost but lacks cluster access
The Reliability Paradox
The core tension the session addressed:
- Teams over-provision to avoid downtime, but over-provisioned clusters are still unreliable, since resources are misallocated rather than right-sized
- When teams do optimize, many tools apply aggressive cost-cutting without regard for reliability, causing outages
- After getting burned once, most teams turn automation off entirely, especially in production, leaving savings on the table
Why downtime looms so large: a widely cited DataBank estimate puts the cost of enterprise software downtime at nearly $9,000 per minute. In microservices architectures, a single failing component (like an ingress controller) can take down the entire environment. See PerfectScale's guide to Kubernetes performance metrics and tuning for more on this.
The Fix: Reliability-First Optimization
Rather than treating cost and reliability as a tradeoff, the speakers argued for optimizing reliability first, which then makes safe, continuous cost savings possible. The goal isn't maximum raw savings, but maximum safe savings.
This approach requires:
- Bidirectional right-sizing: scaling workloads up to prevent incidents, and down to eliminate waste
- Historical workload data and algorithms trained on real production patterns
- Workload awareness: understanding that not all workloads (or even revisions of the same workload) behave the same way, including horizontal vs. vertical scaling behavior
- Guardrails, including revision awareness, rollout awareness, granular custom-resource policies, and maintenance windows, part of PerfectScale's visibility and governance layer
Demo Highlights: PerfectScale in Action
Arthur walked through how the PerfectScale platform applies this approach in practice:
- Risk-first dashboard: high-risk workloads are surfaced immediately, ahead of savings potential (part of Performance Optimization)
- Root-cause visibility: usage graphs show exactly when and why a workload hit OOM kills or throttling
- Revision tracking: every change (manual, automated, or via deployment tools like Argo) is logged against the workload's history, making it easy to see what caused or fixed an issue
- Alerting: risk events surface in-app or via Slack, Teams, or email, via Visibility and Governance
- HPA-aware recommendations: PerfectScale factors in existing horizontal pod autoscaler configs rather than contradicting them (unlike tools such as VPA)
- In-place updates: resource changes can be applied without restarting pods, avoiding service disruption
- Granular automation policies: configurable per cluster, namespace, or workload (e.g., "balanced," "max headroom," "max savings" for dev/test clusters)
- Karpenter integration: node pool level recommendations to improve autoscaling efficiency (see also the latest Karpenter v1.14 update)
- Unevictable pod detection: flags pods (e.g., using local host paths) that block node consolidation and quietly waste spend
- Trends dashboard: tracks cost and risk over time, by cluster, to prove the optimization strategy is working
Q&A Highlights
- On HPA "thrashing": PerfectScale detects HPA triggers and avoids recommendations that conflict with autoscaling behavior.
- On load testing: Load testing is a good way to set initial resource requests/limits, but PerfectScale's recommendations are based on real production utilization patterns. The two are complementary, not a replacement for each other.
- On measuring risk: Risk is scored per workload (e.g., under-provisioned CPU request, under-provisioned memory limit), with severity shown as a simple dot rating (three dots for high risk, two for medium).
Key Takeaways
- Over-provisioning doesn't guarantee reliability. Under-tuned resources can coexist with wasted spend in the same cluster
- Aggressive, reliability-blind cost tools are why most automated optimization gets turned off in production
- A reliability-first approach, combining bidirectional right-sizing with workload-aware guardrails, makes it possible to automate safely instead of choosing between cost and uptime
Ready to see it on your own clusters? Request a PerfectScale demo.
Frequently asked
questions
Why does cost optimization sometimes cause outages in Kubernetes?
Most cost-optimization tools focus purely on reducing resource usage without accounting for how workloads actually behave. They rip out existing autoscaling setups, request broad write access to production, and resize nodes without understanding the risk. The result is OOM kills, throttling, and outages that erase any savings.
What is reliability-first optimization?
It's an approach that addresses reliability risk before pursuing cost savings. Instead of chasing maximum raw savings, the goal is maximum safe savings: workloads are right-sized in both directions, scaled up where they're at risk and scaled down where they're wasteful, so cost reduction never comes at the expense of stability.
What is bidirectional right-sizing?
It's the practice of adjusting resource requests and limits in both directions: increasing them for under-provisioned workloads to prevent incidents, and decreasing them for over-provisioned workloads to eliminate waste. Most cost tools only do the latter.
Does PerfectScale work with HPA and Karpenter?
Yes. PerfectScale is HPA-aware, meaning it factors existing horizontal pod autoscaler configurations into its recommendations rather than contradicting them. It also integrates directly with Karpenter, providing node pool-level recommendations to improve bin-packing and autoscaling efficiency.
Can resource changes be applied without restarting pods?
Yes. PerfectScale supports in-place updates, which apply new resource requests and limits without evicting or restarting the pod, avoiding service disruption.
How does PerfectScale measure risk?
Risk is scored per workload based on signals like under-provisioned CPU requests or under-provisioned memory limits, with severity shown on a simple scale (for example, three dots for high risk, two for medium).
Should I rely on load testing instead of PerfectScale's recommendations?
They work best together. Load testing helps set initial resource requests and limits based on anticipated load. PerfectScale then verifies those assumptions against real production utilization data and continuously fine-tunes recommendations from there.
How much can Kubernetes cost optimization actually save?
Unoptimized production clusters typically waste $5,000–$10,000 per month, and 80% of unoptimized clusters are over-provisioned. Right-sizing workloads and node pools can eliminate most of that waste without compromising reliability.