1. Handling Deterministic Faults
For faults that have a specific handling procedure, just let the Agent handle them and send a notification to spread the word.
For similar automated handling, we have exception load deletion at the application layer, disk cleanup at the node layer, masking an offline GPU card, masking a node, and so on. First, find the things the team encounters most often and that consume the most manpower, and automate those. You can think of the Agent here as LLM + SOP: the model helps you pick the right standard procedure to handle a standard fault and get a standard result.
2. Inspections That Sense Risk
After finishing the previous step, I paused for a long time, filling in some of the team’s observability capabilities, and at the same time exported the five dimensions of capability β metrics, logs, events, traces, and standard operations β as a unified standard protocol through the ops-mcp-server project.
Keeping a dozen clusters and several hundred nodes running stably, and quickly discovering, locating, and fixing faults, is the foundation of my job. Once the numbers reach a certain level, things that used to happen only by chance gradually become inevitable. There are many faults to troubleshoot every day, and heterogeneous GPU scenarios plus an AI architecture that unifies training and inference significantly increase the difficulty of root cause analysis.
While a fault is happening, you frequently switch back and forth across many pages looking for the root cause. The inspection Agent exists to solve exactly this problem. At this point the Agent is not AI-driven. Writing an AI Agent and reaching for too much AI capability too early only gets you a local optimum, and it may not be able to keep evolving afterward.
But that does not mean you cannot put AI inside it. You can first write many inspection Agents, and let the AI choose which inspection Agents need to be enabled based on the scenario.
3. Attempting to Analyze Faults
Open-ended root cause analysis is where AI Agents truly shine. With the groundwork already laid β you have established a number of standard fault handling procedures and can also run all kinds of inspections on clusters, nodes, middleware, and networks β the rest is left to the AI to analyze.
Analyzing hot faults is undoubtedly of enormous value; being able to discover, locate, and fix a fault first in the team is a very gratifying thing.
The key point of automated fault analysis is associating resources. Unfortunately, you may be like us and have no such database at all β it is all passed on by word of mouth, all looked up in real time during analysis. Putting a lot of effort into filling in that database leaves consistency of later maintenance as another problem. What to do? Hand it to the LLM.
Earlier we built many inspection Agents to query resources; give the queried information to the AI and it can extract the associated information. Query continuously, analyze continuously, until a satisfying root cause is found.
The figure above is the root cause analysis result I obtained automatically after an alert, based on the most primitive Prometheus and Elasticsearch, without any Trace instrumentation and without any intrusion.
