Troubleshooting Kubernetes with AI automates the identification, analysis, and remediation of complex cluster failures, shifting incident response from manual log sifting to AI-guided resolution. Rather than spending hours running kubectl commands by hand, DevOps teams use AI agents to scan configurations, correlate cross-layer signals, and fix problems in a fraction of the time.
In my previous post I showed how to build Kaia (the Kubernetes AI Agent) using Autogen 0.4 and Ollama. This applied the naive and simple approach of accessing kubectl using python’s subprocess and AutoGen’s FunctionTool . But nowadays the consensus seems to be that agent interactions should be standardized and the leading standard for that right now is MCP (Model Context Protocol). (For more details on Model Context Protocol - check out our recent webinar with Patrick Debois)
AI-powered troubleshooting platforms do not just read raw logs, they parse the entire state of your infrastructure:
- Automated detection: A Kubernetes controller or observability tool watches warning events such as FailedScheduling or CrashLoopBackOff.
- Contextual analysis: The AI agent queries cluster state, maps dependencies, and ingests pod logs, metrics, and events.
- Natural language explanation: Instead of throwing raw YAML errors, the model translates the failure into a human-readable diagnosis.
- Remediation and validation: The AI proposes a precise fix and checks whether the application returns to a healthy state.
The tooling for AI-driven Kubernetes troubleshooting falls into a few main categories:
- Open-source diagnostics: command-line and in-cluster tools such as K8sGPT, kubectl-ai, and kagent.
- The Model Context Protocol (MCP): the emerging standard that securely connects agents and IDEs to your cluster context.
- Cloud-native and enterprise assistants: managed AI layers baked into GKE, EKS, AKS, and observability suites.
The pace here is relentless. Agent frameworks now treat MCP as a built-in standard, open-source diagnostic tools keep adding analyzers and model backends, and the major clouds have turned AI troubleshooting into a native feature of their managed Kubernetes offerings. This post walks through building a hands-on agent, then situates it against that broader, fast-moving toolkit.
MCP in a nutshell
Model Context Protocol is a standard developed by Anthropic for connecting AI models with data sources. In a typical MCP architecture the AI assistant or agent is the client consuming the services of an MCP server. The MCP server provides any or all of the following features:
- Resources: Context and data, for the user or the AI model to use
- Prompts: Templated messages and workflows for users
- Tools: Functions for the AI model to execute
The following diagram shows how an MCP server provides AI agents with capabilities in a typical agentic architecture:
Beside just transferring the LLM-induced calls to the external tool, the protocol also allows an MCP server to preserve some context and enrich the original prompts.
MCP for Kubernetes
Since getting released, the MCP protocol has seen significant uptake from the community. All major cloud services providers have already published MCP server implementations for their public API and open-source enthusiasts were quick to implement MCP for their favorite tools. And of course Kubernetes was no exception. One of the most complete MCP server implementations for K8s comes from Alexey Ledenev - a senior cloud architect at DoiT International. Beside supporting kubectl it also wraps around such important modern kube-related utilities as helm, argocd and istioctl.
That uptake has only accelerated. MCP has since crossed more than two hundred server implementations, and the wider protocol layer has consolidated, with the complementary agent-to-agent (A2A) protocol now governed by the Linux Foundation. Native MCP support has effectively become table stakes for every serious agent framework.
The open-source AI troubleshooting ecosystem
The K8s MCP server is one piece of a fast-growing ecosystem of open-source tools that bring AI to cluster operations. Alongside it, a handful of targeted utilities have become the community standards for AI-assisted diagnostics.
K8sGPT
K8sGPT is a CNCF Sandbox project that scans your cluster for anomalies. Using the k8sgpt analyze --explain command, it pairs built-in analyzers with an LLM backend to pinpoint issues like misconfigured services or broken ingress rules and explain them in plain English. It works with multiple backends, from OpenAI and Azure OpenAI to local models via Ollama, so telemetry does not have to leave your infrastructure.
kubectl-ai
kubectl-ai is a Google-backed tool that adds a natural language interface on top of kubectl. Instead of remembering exact syntax, you describe what you want, for example scaling a deployment to five replicas in staging, and it translates the request into the corresponding commands.
kagent
kagent is an open-source, Kubernetes-native agent framework created by Solo.io and now a CNCF Sandbox project. It acts like a junior DevOps engineer that lives inside the cluster, running sequential checks to isolate problems such as an unreachable application. Where K8sGPT diagnoses and kubectl-ai helps you execute, kagent autonomously operates the cluster.
The official repo shows a couple of examples integrating the MCP server with Claude Desktop for troubleshooting and application deployment. But what’s the fun in using AI if it’s not agentic, right? So I deided to see how this integrates with one of the popular agentic frameworks. Initially I thought of going with AutoGen 0.4 (as I did in my previous experiment), but I quickly found out that the MCP support in AutoGen looks bolted on as an afterthought via autogen_ext.tools.mcp and feels clunky at best (For some reason - the MCP server container started by AutoGen would exit before the agent had a chance to talk to it).
Therefore I decided to take another agentic framework for a test drive - namely PydanticAI that comes from the same team that built Pydantic - the data validation library that FastAPI is based on. According to the official website : “PydanticAI is a Python agent framework designed to make it less painful to build production grade applications with Generative AI.” I instantly liked the “less painful” part. And in fact - integrating the MCP server with my agent’s code was as easy as:
- Starting the MCP server in a container (note how I’m mounting my .kube./config into the container so it gets access to whatever cluster I have in my current context) :
server = MCPServerStdio( 'docker', args=[ 'run', '--network=host', '-i', '-v', f'{os.path.expanduser("~")}/.kube:/home/appuser/.kube:ro', #mount .kube/config 'ghcr.io/alexei-led/k8s-mcp-server:latest' ])- Providing it to the agent in the mcp_servers list:
agent = Agent( model, mcp_servers=[server], system_prompt=system_prompt, retries=5)An attentive reader will notice that agent initialization requires 2 more parameters: the model and the system prompt.
Multiple model providers
In my previous experiment I had the goal of running the whole setup self-hosted and open-source. But the results from the models I’ve tried with Ollama were quite discouraging - it looks like the smaller local models aren’t very good with using MCP. So this time I decided to support multiple model providers - with Ollama still being the default.
I added support for Gemini and Github models as both provide some free requests. In my tests - openai/gtp-4o from Gtihub models delivered the best results and it also has the added benefit of zero setup - all one needs is their Github PAT (Personal Access Token) to get this started.
def create_model(model_provider: str): """Create model based on provider choice.""" if model_provider == 'ollama': # Ollama using OpenAI-compatible API return OpenAIModel( model_name=os.environ.get('OLLAMA_MODEL_NAME', 'llama2'), provider=OpenAIProvider(base_url='http://localhost:11434/v1') ) elif model_provider == 'gemini': # Google Gemini return GeminiModel( model_name=os.environ.get('GEMINI_MODEL_NAME', 'gemini-2.0-flash') ) elif model_provider == 'github': # GitHub models using OpenAI-compatible API github_token = os.environ.get('GITHUB_TOKEN') if github_token: os.environ['OPENAI_API_KEY'] = github_token return OpenAIModel( model_name=os.environ.get('GITHUB_MODEL_NAME', 'openai/gpt-4o'), provider=OpenAIProvider(base_url='https://models.github.ai/inference') ) else: raise ValueError(f"Unsupported provider: {model_provider}. Choose from: ollama, gemini, github")Note that while there’s a separate GeminiModel interface for Google Gemini - the access to Ollama and Github models is still provided via the OpenAIModel interface. I’m guessing this will change in the follow-up PydanticAI releases if somebody cares enough to provide custom interfaces for them.
Since this experiment, PydanticAI has moved quickly. A V2 release brought a harness-first redesign, and MCP is now a first-class, built-in feature rather than something bolted on, so connecting an agent to an MCP server takes even less plumbing than the snippets above suggest.
Cloud-native and enterprise AI solutions
Beyond open-source tooling and self-hosted agents like Kaia, every major cloud provider and monitoring suite now bakes AI diagnostic layers natively into their managed environments. If you would rather not wire up your own agent, these can be a fast on-ramp:
The System Prompt
I started out with a prompt I crafted myself. It wasn’t working very well, so I asked Claude to improve it for me.
My original version was:
You are a Kubernetes operations assistant. You have access to a Kubernetes cluster through an MCP server.
To interact with the cluster, you must use the tools/call method with the following format:
For kubectl commands:
{ "jsonrpc": "2.0", "method": "tools/call", "params": { "name": "execute_kubectl", "arguments": { "command": "<your-kubectl-command>" } }}Always use the MCP server commands to execute tasks. Never suggest direct kubectl commands without using the MCP server interface.
Before executing commands, make sure to validate them and consider their impact.
Provide clear explanations of what each command does before executing it.
Never chain commands together in a single request.
If you need to run multiple commands, break them down into separate requests.
If any command fails - retry taking into account the error message.
The prompt that Claude came up with is so long that I decided to omit it here. You can see it in the code. Does it work better? I think so. Haven’t really benchmarked the results for accuracy or speed.
Common scenarios resolved by AI
Once an agent like Kaia is wired to the cluster, the kinds of problems it handles well tend to share a pattern, they are tedious to trace by hand but obvious once the full context is assembled:
- Nested configuration errors: tracing a failed Helm deployment where a values file points to a template that references a missing key in a ConfigMap.
- Silent label mismatches: finding bugs where a Service selector looks for a specific label, but the active pods were deployed with a typo in theirs.
- Ephemeral context salvaging: capturing logs, host counters, and traces before a failing pod crashes and wipes out the debugging evidence.
Crucial security guardrails
Running AI over production clusters introduces real security and compliance considerations. To keep risk in check, most frameworks (and the way I mounted a read-only kubeconfig for Kaia) lean on a few architectural guardrails:
- Read-only permissions: most diagnostic tools need only read-only cluster RBAC. The AI suggests a fix, but a human executes or approves the command.
- GitOps alignment: rather than mutating cluster state directly, advanced agents open a pull request, which keeps a clean audit log and prevents configuration drift.
- Data masking and privacy: tools like K8sGPT strip out secrets, passwords, and PII locally before sending telemetry to external LLM providers.
Seeing the new Kaia in action.
I still can’t decide what’s more fun - building agents or running them?
Where AI-driven Kubernetes troubleshooting is heading
The gap between a weekend project like Kaia and production tooling is closing fast. Cloud providers have made proactive, multi-turn troubleshooting agents a headline feature, with recent releases wiring assistants directly into gcloud, kubectl, and Terraform so they can investigate and resolve incidents rather than just describe them.
The open-source side is maturing in parallel. Diagnostic tools now ship with twenty or more built-in analyzers and multi-LLM backends, and Kubernetes-native agents run as long-lived custom resources inside the cluster. The practical takeaway is that self-hosted agents and managed assistants are converging on the same pattern: read cluster state through a governed interface, reason over it, and propose an auditable fix.
Anyway - to see the new Kaia in action - watch this video. Then clone the repo, try to run this for yourself and let me know how it works for you.
Happy hacking!