Solidus Labs Reduces Kubernetes Resilience Issues by 90%

Solidus Labs and Develeap deployed PerfectScale by DoiT to automate right-scaling of pod resources across their Kubernetes environment. PerfectScale continuously analyzes workload behavior and provides precise resource recommendations, eliminating the cycle of manual adjustments. It also delivers evidence-based recommendations directly to service owners, accelerating remediation, and surfaces cost-optimization opportunities so unused resources can be reallocated to clusters that need more capacity.

Solidus Labs is on a mission to enable safer crypto trading throughout the investment journey across all centralized and DeFi markets. As the founder of industry-leading initiatives, Solidus is deeply committed to ushering in the financial markets of tomorrow. To support rapid growth in crypto markets and meet ever-increasing client demand, Solidus leverages Amazon EKS as the foundation of its application infrastructure. The company partners with Develeap, one of the largest DevOps consultancies in Israel, which built the initial architecture and provides ongoing support, monitoring, observability, alerting, and cost optimization — helping Solidus scale to a dozen multi-regional clusters serving clients across the globe.

The Develeap team implemented many capabilities to keep the Kubernetes environment running smoothly, including Keda for horizontal pod autoscaling. However, they lacked the ability to optimally right-scale pod resources, leading to CPU throttling and OOM issues. The environment was in constant flux: 'R&D is releasing changes on an hourly basis, and, due to the nature of our business, some of our clients send data in large batches, while others use us as a real-time service, making it hard to predict the load fluctuations on our services,' said Ben Hoffman, R&D Director of Solidus Labs. The team spent hours stabilizing their biggest cluster and replicating capacity to others, but results were short-lived and created waste across smaller clusters.

Shemtov Fisher, DevOps Engineer at Solidus Labs/Develeap, described the manual cycle: 'I jumped into a Grafana and pulled in metrics from Prometheus and logs from Logz.io, and made adjustments to the requests based on the different peaks of our environment. Then a few weeks would pass, and we started seeing throttling and memory issues resurface, leading to a second round of adjustments. When I jumped in a third time, I knew we needed a solution in place to help automate this process. PerfectScale by DoiT is the exact solution we needed to fill this gap.'

Shortly after implementing PerfectScale by DoiT, Solidus was able to proactively right-scale pod resources, leading to a significant reduction in CPU throttling and OOM issues. 'We went from multiple issues a day, to maybe one or two issues in the last month,' said Hoffman. 'With PerfectScale, we have seen over a 90% reduction helping us ensure our applications have the capacity to meet our customer demand.' PerfectScale also drastically reduced MTTR for capacity-related issues. As Barak Arzuan, DevOps Engineer at Solidus Labs/Develeap, explained: 'Before PerfectScale, the DevOps team would get an alert when an issue occurred, then we would triage the issue to the proper service owner to resolve. Depending on the criticality, it could take hours or even more for the service owners to evaluate the issue and provide us with the proper resource requirements. With PerfectScale, we can immediately provide the service providers with evidence on why the issue is happening along with precise recommendations on how to resolve it.'

Adding capacity to improve resilience typically comes with a price, but the Develeap team leveraged PerfectScale's cost-optimization capabilities to move unused resources to areas that needed additional capacity. 'In some of our clusters, we found significant cost savings opportunities,' explained Arzuan. 'We were able to reinvest these savings into our clusters that were lacking resources. This resulted in a fully stable, resilient, and cost-effective environment with no impacts on our budget.' Hoffman summarized the impact: 'We have a large number of clients, each using our application slightly differently. Keeping our Kubernetes environment optimized is essential for Solidus Labs to ensure our applications have the resources they need to support our customers today, and as our company continues to grow in the future. PerfectScale is removing time-consuming manual tasks we have faced in the past, making it easy to continuously maintain our system's health and cost-effectiveness.'