.png)
AWS Karpenter is a powerful tool offered by Amazon Web Services that allows you to efficiently manage and optimize your Kubernetes clusters.
Kubernetes (K8s) has become the de facto standard for container orchestration, providing a powerful platform for deploying and scaling containerized applications. However, managing Kubernetes clusters can be challenging, especially when it comes to optimizing resource utilization and cost efficiency. This is where Karpenter, a Kubernetes-native autoscaling solution, comes into play. Karpenter makes it easy to deploy, manage and automatically scale Kubernetes nodes.
In this article, we will explore how to leverage Karpenter Profiles to take advantage of AWS Reserved Instances (RIs), which can significantly reduce your AWS bills. Specifically, we will show you how to ensure that the correct instance type is deployed first and how to deploy different types of reserved instances to maximize the benefit of your reserved instances. By following the steps in this article, you will be able to deploy and manage your Kubernetes cluster on AWS in a cost-efficient manner.
Why is this important? Because it is very easy to get a situation when you ordered a reserved instance but it goes unused and becomes a waste of money. For example, say you have (5) m5.xlarge RI in us-east-1a zone and your Karpetner configuration looks like :
As Karpenter has it's own mechanism of choosing the right instance type and zone for scaling a new node, the chance that it will scale (5) m5.xlarge in zone us-east-1a is very low.
Understanding AWS Karpenter
Before diving into the details, let's take a moment to understand what AWS Karpenter is and what it can do for you. AWS Karpenter is an open-source, self-managing autonomous scaling controller for Kubernetes. It automates the provisioning and termination of worker nodes in your Kubernetes cluster, ensuring optimal resource utilization and cost efficiency.
What is AWS Karpenter?
AWS Karpenter, powered by the Kubernetes Horizontal Pod Autoscaler (HPA), helps you dynamically scale your cluster's capacity based on the compute demands of your workloads. It automatically adjusts the number of worker nodes in your cluster, keeping up with the workload requirements and scaling down during periods of low demand.
Key Features of AWS Karpenter
There are several key features that make AWS Karpenter a valuable tool for managing your Kubernetes clusters:
- Autoscaling: AWS Karpenter automatically scales the capacity of your cluster based on workload demands, ensuring efficient resource utilization.
- Cost Optimization: By dynamically adjusting the number of worker nodes, AWS Karpenter helps you optimize costs by minimizing underutilization.
- Self-Healing: AWS Karpenter monitors the health of your worker nodes and automatically replaces any failed instances, ensuring the availability and reliability of your cluster.
- Integration with AWS EKS: AWS Karpenter seamlessly integrates with Amazon Elastic Kubernetes Service (EKS), providing a streamlined experience for managing your Kubernetes workloads.
Benefits of Using AWS Karpenter
The benefits of using AWS Karpenter are numerous:
- Efficient Resource Utilization: With AWS Karpenter, you can maximize resource utilization by automatically scaling your cluster up and down based on workload demands.
- Cost Savings: By optimizing resource allocation, AWS Karpenter helps you reduce infrastructure costs and eliminate overprovisioning.
- Simplified Management: AWS Karpenter automates the provisioning and termination of worker nodes, saving you time and effort in managing your cluster's capacity.
- Improved Cluster Performance: AWS Karpenter ensures that your cluster is always right-sized, resulting in better performance and responsiveness for your applications.
- AWS Karpenter provides advanced monitoring capabilities, allowing you to gain insights into the performance and utilization of your Kubernetes cluster. With detailed metrics and logs, you can easily identify bottlenecks, optimize resource allocation, and troubleshoot any issues that may arise.
- Seamless integration with other AWS services. By leveraging the power of AWS, you can easily extend the capabilities of your Kubernetes cluster. For example, you can integrate AWS Lambda functions to automate certain tasks, or leverage Amazon RDS for managed database services.
- AWS Karpenter offers a high level of flexibility and customization. You can define custom scaling policies based on your specific requirements, allowing you to fine-tune the scaling behavior of your cluster. Whether you need to prioritize cost savings, performance, or a balance between the two, AWS Karpenter gives you the control to achieve your desired outcomes.
In conclusion, AWS Karpenter is a powerful tool that simplifies the management of Kubernetes clusters, automates scaling, optimizes resource utilization, and enhances the performance of your applications. By leveraging AWS Karpenter, you can focus on developing and deploying your applications, while leaving the management of your Kubernetes infrastructure to the autonomous scaling controller.
Setting Up AWS Karpenter
- Order (2) reserved instances.
- Create (3) Karpenter profiles, (2) for AWS Reserved Instances and (1) default for stateful workloads.
- Deploy some pods to check and make sure that Karpenter scales the correct instance types.
Before we begin, there are a few prerequisites that must be in place.
- We need to have a working installation of Karpenter and a basic understanding of what a Karpenter Provisioner is and how it works. If you haven't installed Karpenter already, you can follow the official documentation to do so.
- And we need to have access to our AWS EC2 Console to order reserved instances.
1. Configuring AWS
IMPORTANT NOTE: Once you order a reserved instance you automatically commit that you will pay for it, with durations ranging from 12 to 36 month. Be very careful and double check all settings to make sure you are purchasing for the right instances.
For this test, I decided to buy (2) m5.xlarge instances, one in zone us-east-1b and second in zone us-east-1c (small recommendation: check the zones of your volumes and use the same zones for AWS Reserved Instances, this insures K8s will be able schedule pods on RI nodes).
- Configure all settings for the AWS Reserved Instances for us-east-1b.
- Press “Search”, choose an instance and press “Add to Card“.
- Do the same for the us-east-1c zone and press “View cart“.
- Select all instances and press “Order all“.
- Now we can see our AWS Reserved Instances in the console.
2. That all with AWS, let's move to the Karpenter.
For the demo we will configure three provisioners, first will be default, second for us-east-1b and third for us-east-1c. Below are 3 files with provisioner configuration (Note: This is not all the possible settings, you can find more in the Karpenter provisioner configuration.)
stateful.yaml:
stateful-reserved-us-east-1b.yaml
stateful-reserved-us-east-1c.yaml
NOTE: Every environment configuration can be different. I recommend you to read all articles, check your environment and only then configure and apply everything.
Let's go through the configuration:
- consolidation - This attempts to reduce cluster cost by both removing unneeded nodes and down-sizing them. I disabled this for AWS Reserved Instances provisioners but enabled for default. When it is enabled, Karpenter can drain the node and all pods will be evicted from the node, so please choose the correct policy for your cluster.
- kubeletConfiguration - Additional Kubelet args, it is very important to reserve some resources for system and kubelet service.
- labels - This is a label that will be added to the node. In this demo I use type: stateful, meaning the node(s) will be for stateful workloads (you can use any labels that you want) and an additional label billing: reserved for future analytics or node selector.
- limits - General limits prevent Karpenter from creating new instances once the limit is exceeded. For AWS Reserved Instances set the same resources that m5.xlarge has (4 cores and 16 GB memory).
- provider - This is a cloud provider-specific custom resource, you should change it according to your cluster.
- requirements - These are the requirements that constrain the parameters of provisioned nodes.node.kubernetes.io/instance-type - for the default stateful provisioner I use many instance types and allow Karpenter to choose correct type but for AWS Reserved Instances only m5.xlarge.topology.kubernetes.io/zone (the default stateful provisioner) I did not set any zone, meaning that Karpenter will decide what zone to use for the new node but for AWS Reserved Instances use the zone selected when I purchased the reserved instance.
- taint - Provisioned nodes will have these taints.
- weight - This is one of the most important settings for reserved instances. It sets the priority given to the provisioner when the scheduler considers which provisioner to select. Higher weights indicate higher priority when comparing provisioners. AWS Reserved Instances provisioners have a higher weight (20) so karpenet will try scale nodes from these provisioners. If there is no capacity, use a default provisioner that has lower weight (10).
Applying configuration:
Now we are going to check if everything works as expected.
I am creating deployment with 2 replicas:
We can see that they are in a pending state now:
And Karpenter started 2 new nodes:
Let's check Karpenters logs:
From logs we see that Karpenter launched 2 new instances from reserved instance profiles. Let's describe the instance and check labels:
But what happens if we scale more replicas? We can easily check it:
Now 2 additional pods are in a pending state:
And what did Karpenter say? It launched 2 new instances but used a default stateful provisioner:
That's cool, everything works as expected!!! Do not forget to delete test deployment.
In conclusion, leveraging AWS Reserved Instances can provide significant cost savings for your Kubernetes cluster on AWS and using Karpenter can help you maximize those savings. Karpenter ensures that the correct instance type is deployed first and allows you to deploy different types of reserved instances. However, it's important to monitor your usage of reserved instances and to make adjustments to your Karpenter Profiles as needed to ensure you're maximizing your cost savings. With the right tools and strategies in place, you can deploy a high-performing and cost-efficient Kubernetes cluster on AWS with ease.
BONUS:
If you are using Prometheus Operator and collect Karpenter's metrics, you can add a simple alert to check if all reserved instances that we have are utilized. Karpenter provide many metrics that can be collected, but unfortunately there are no metric that show instance count by type, so I decided use metric karpenter_pods_state it show pod status on the node that was provisioned by Karpenter. Every node in the Kubernetes cluster has kube-proxy pod and we can use it for alerts.
In the first part of expr: sum(karpenter_pods_state{capacity_type="on-demand", instance_type="m5.xlarge", name=~"kube-proxy.*", phase="Running", zone="us-east-1b"}) by (zone,instance_type,capacity_type) < 1 we check that in zone us-east-1b not less than 1(set it to your AWS Reserved Instance counts) m5.xlarge instances.
If for example in the cluster we do not have any m5.xlarge above expression will return “Empty query result“, so we check it in the second part of expr: absent(karpenter_pods_state{capacity_type="on-demand", instance_type="m5.xlarge", name=~"kube-proxy.*", phase="Running", zone="us-east-1b"}) == 1.
The final PrometheusRule file will be:
.png)
.png)
