1. Keep Trying to Ship the AI Application Side

Based on my understanding of operations, I developed an open-source operations tool, https://github.com/shaowenchen/ops .

The Ops tool divides operations work into two categories — script execution and file distribution — and both operations targets, hosts and Kubernetes clusters, implement both of these operations.

The capabilities Ops exposes externally are the Ops Cli command-line terminal, the Ops Server API endpoints, Ops Controller for cluster-side resource management, and a simple UI.

Although I developed Ops Copilot half a year ago, it does not bring out the capabilities of the Ops project. It is very standalone — merely connected to an LLM, able to provide some conversation and to execute scripts automatically on the local machine, nothing more. I hope there will be a chance later to rewrite Ops Copilot along this AI Agent technical direction.

Lately my main work goal has been to build an AI Agent on top of Ops that automates some operations work.

2. Finding an Iterable AI Agent Technical Path Matters

AI Agent is an important capability that lets AI interact directly with the physical world — the key technology that can genuinely raise productivity and free humans from repetitive, mechanical work.

AI Agent is wonderful, but that is from the perspective of a technology consumer. From the perspective of an AI Agent developer, the topic is far less comfortable:

1. A strong demand, ideally matching your job content, that drives you to develop an AI Agent
2. Scenarios to polish the AI Agent — it takes a large number of use cases to refine and continuously optimize the implementation
3. An iterable technical path

1 and 2 depend on the fundamentals of your position — whether it supports you in developing an AI Agent. 3 is within your own control, and this post focuses on the iterable AI Agent technical path.

Introducing new technology into a team is never done in one step; it always comes with periodic reporting, periodic results, risk control, and so on. Even if you have good ideas and execution ability, an internet team can hardly tolerate too long a stretch of manpower investment with no visible results.

The internet product development model has already accepted agile thinking: continuously build usable products, deliver them, get feedback, and keep iterating.

3. What a Service-Oriented AI Agent Is

“Service-oriented AI Agent” is a term I coined; let me introduce it below.

3.1 The Difference Between an AI Agent and a Copilot

An AI Agent emphasizes autonomy — being able to solve problems proactively; a Copilot emphasizes assistance and suggestions, and requires human guidance.

By identifying who leads a matter, you can quickly tell the two apart. If the program leads and takes responsibility, it is an AI Agent; if the human leads, it is a Copilot.

3.2 The Service-Oriented AI Agent

Usually when people discuss AI Agents they mean localized ones — ones that need to run on the device being operated, and that can even deploy an LLM service locally when necessary.

As shown above, a service-oriented Agent provides task execution and environment perception capabilities in the form of APIs.

For example, a Python script generated on the AI side must call the Agent’s API in order to be executed, and then returns the execution result. Likewise, if the AI needs the current environment information, it must call the Agent API to get it.

Of course you can also take a multi-Agent approach, as shown above: each Agent connects to the targets it is responsible for, keeping environment perception and execution capability on the device being operated.

A service-oriented AI Agent has the following advantages:

• Separation of duties: the Agent is only responsible for execution and perception, while the AI is only responsible for analysis and aggregation
• More flexible execution environments: different execution environments can be configured based on resources, security, and other factors — for example an Agent for cluster operations, an Agent for GPU operations, an Agent for cloud resource operations
• Easier integration with existing systems: you can develop a SideCar Agent per system to connect to the AI, instead of one localized AI Agent integrating every system

3.3 Ops Is a Service-Oriented Agent Side

Ops Server is positioned to execute the Task jobs built into Ops through API endpoints. These Tasks are all sensitive operations actions.

Through the Ops Server API endpoints, the following capabilities are available:

Perception:

• Host list
• Kubernetes cluster information
• Information obtainable via Shell inside a Host
• Information obtainable via kubectl for a Kubernetes cluster, plus information obtainable via Shell on each node

Execution:

• Host operations
• kubectl operations on a Kubernetes cluster, plus Shell operations on each node

These capabilities are already enough for the AI to complete most operations tasks.

4 Start Writing a Service-Oriented AI Agent

4.1 Abstracting Two Objects

As shown above, in a service-oriented AI Agent:

• The object that drives the AI is a Pipeline

One Pipeline on the AI side corresponds to multiple Tasks on the Agent side; these Tasks, through a certain ordered arrangement and combination, conditional control, and variable values, achieve the Pipeline’s task objective.

• The object that drives the Agent is a Task

In Ops, I already provide a Task CRD object with a large number of built-in Shell scripts, used to converge and reuse basic operations actions.

4.2 Using Tasks to Encapsulate the Agent’s Basic Operations

These are some of the operations Tasks we use in production:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

kubectl -n ops-system get task

NAME                        CRONTAB   STARTTIME   RUNSTATUS
check-gpu-drop                                    init
check-node-existed                                init
check-npu-drop                                    init
check-pod-existed                     36m         successed
check-svc-existed                                 init
cordon-node                                       init
delete-pod                            48m         successed
get-gpu-status                                    init
get-os-status                                     init
get-pod-error-logs-byname             36m         successed
get-pod-error-logs-bysvc              62m         successed
get-podlogs-byname                                init
get-podlogs-bysvc                                 init
inspect-clusterip                     120m        successed
list-clusters                         45h         failed
list-error-events-bypod               36m         successed
list-error-events-bysvc               62m         successed
list-events-bypod                                 init
list-events-bysvc                                 init
list-nodes                                        init
list-pods                                         init
list-pods-abnormal                                init
list-pods-bysvc                       62m         successed
list-tasks                            45h         successed
none-action                                       init
top-nodes-bycpu                                   init
top-nodes-bymem                                   init
top-pods-bycpu                                    init
top-pods-bycpu-node                               init
top-pods-bymem                                    init
top-pods-bymem-node                               init
uncordon-node                                     init

Adding a new Task only requires writing a Yaml file; developing a new Task is very easy.

4.3 Using Pipelines to Connect Business Scenarios

Since Ops does not abstract the concept of a Pipeline, I defined the Pipeline object on the AI side.

For each specific scenario, you can write a Pipeline; here is an example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

var allPipelines = []ops.LLMPipeline{
	pipelineGetClusterIP,
}

// 查看 cluster ip 详情
var pipelineGetClusterIP = ops.LLMPipeline{
	Desc:      "Query - 查看 cluster ip 详情",
	Namespace: "ops-system",
	Name:      "get-cluster-ip",
	NodeName:  "anymaster",
	Variables: []ops.VariablePair{
		{
			Key:      "clusterip",
			Desc:     "For example, `clusterip: 1.2.2.4`, 1.2.2.4 is clusterip",
			Required: true,
		},
	},
	LLMTasks: []ops.LLMTask{
		{
			Name: "inspect-clusterip",
		},
		{
			Name: "app-summary",
		},
	},
}

Two Tasks are used here.

• inspect-clusterip - queries cluster ip details; called through the API, the task is dispatched to the Agent for execution
• app-summary - summarizes the output information via the LLM.

There are two kinds of Tasks: one is retrieved from the Agent in real time; the other is a Task custom-defined on the AI side, meant to encapsulate some tasks unrelated to Ops — though of course you could also develop another Agent to execute such tasks.

4.4 Development on the AI Side

Usually an AI Agent is an infinite-loop process that continuously processes expected data, then makes a plan, executes the plan, reflects, and makes a plan again — round and round until the problem is solved.

But this approach consumes a lot of Tokens, and in the early days of project development, the Prompt, Task, and environment perception are not yet well prepared, so it is hard to get good results. Therefore, it is mainly divided into the following iteration stages:

1. Implement AI Agent assistance for preset scenarios

For a specific scenario with a fixed context and determined input, our goal in developing the AI Agent is very clear, and it is easy to get good results. At the same time, it lets us accumulate the relevant Task capabilities and debug the Prompt.

2. Recall key information through a vector store and surrounding systems

It is foreseeable that as the number of onboarded scenarios grows sharply, the Pipeline descriptions, cluster, and host information that must be provided to the LLM each time will increase, leading to the following problems:

• Overly long text reduces the accuracy of identifying which Pipeline to execute
• Insufficient context and feedback cause the AI Agent to be unable to solve a problem or verify whether it has been solved

At this point you need data that is more relevant to the input, and more of it — recalling Pipeline-related information, execution results, and information from the surrounding systems it depends on. Let the AI Agent, based on this information, autonomously execute Pipelines, get results, and keep executing until the termination condition is reached.

3. Autonomous wake-up

There is no longer a need for a human to wake it up with an at mention; instead the AI Agent can automatically capture key information and wake itself. Technically this is not difficult; the difficulty lies only in whether the AI Agent is ready and able to adapt to the full set of scenarios.

Waking it up with an at mention can only cover part of the scenarios, but if you let the AI Agent wake itself, there are many more details and boundaries to consider.

4. The AI writes Pipelines autonomously

From the input, the AI Agent autonomously searches the vector store, gets the relevant Tasks and context information, and writes Pipelines — even new Tasks. Only then has the AI Agent reached the ideal state.

4.5 A Look at What I Have Achieved

Following the approach above, I have currently completed up to stage 1, but 2, 3, and 4 are technically feasible and need time to iterate.

• Restarting a Pod

• Viewing the details of an alert

I have basically achieved the connection between Pipeline and Task, and can handle some basic operations scenarios. Notably, the cost of onboarding a new scenario is especially low. You only need to reuse or add Tasks and register them to allTasks, and compose a Pipeline and register it to allPipelines, and a new scenario can be added.

5. Alignment Is a Huge Challenge in AI Agent Development

Perhaps not only in AI Agent development — all LLM application scenarios run into the alignment problem.

5.1 How to Effectively Extract Key Information

When the AI side runs a Pipeline, it needs to match the Pipeline and extract the relevant run parameters. To that end, the Pipeline needs:

• A detailed description of the Pipeline’s purpose

1
2
3

var pipelineGrafanaAlertPodErrorRequest = ops.LLMPipeline{
	Desc:      "Analysis - Grafana告警Pod异常请求比例大于10%",
}

Desc lets the LLM accurately identify the Pipeline’s purpose.

• A detailed description of variable meanings, format characteristics, etc.

1
2
3
4
5
6
7
8
9

type VariablePair struct {
	Key          string
	Value        UniversalValue
	Desc         string
	Regx         string
	ExampleValue UniversalValue
	DefaultValue UniversalValue
	Required     bool
}

Inaccurate variable extraction is a common problem, so the variables it depends on need detailed descriptions. These descriptions include the variable’s description, regular expression, example value, default value, whether it is required, and so on — the more and the more distinctive the features, the easier it is for the LLM to make the right judgment.

To let the AI side correctly identify which cluster object is being operated on, I even submitted the dynamically obtained Cluster list directly to the LLM.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

parmerters := jsonschema.Definition{
		Type: "object",
		Properties: map[string]jsonschema.Definition{
			"nameRef": {
				Type:        "string",
				Description: m.GetClusterManager().BuildtMarkdown(),
				Enum:        m.GetClusterManager().GetList(),
			},
		},
		Required: []string{"nameRef"},
	}
}

5.2 Retry Still Works

In one month I wrote three or four versions of the AI Agent. In the first version the AI connected directly to Tasks with no Pipeline abstraction; it worked alright, but this situation came up often.

As shown above, sometimes the LLM does not output the Json-format data I want, but instead outputs a Plan directly. The expected response would be for the LLM to return a piece of Json data, then for the Agent to execute those Tasks and return the final result to the user. The figure below is an example that meets expectations:

Therefore Retry remains one of the key methods for effectively solving the alignment problem. When the LLM does not understand your intent, you just need to retry once. When the LLM cannot effectively extract key information, you just need to retry once. When the LLM cannot give the expected result, you just need to retry once.

When the program enters an unexpected state while executing a Pipeline, a retry can basically improve the final response immediately.

5.3 Reflection\Reward Improves Accuracy

In the same Retry loop, if you can add a bit of feedback or reward, it also improves the LLM’s ability to align with the scenario’s intent.

There is a relevant description at https://blog.langchain.dev/reflection-agents/ , as shown below:

We only need to check whether the Pipeline’s state meets expectations, state our own requirements, check certain key states, give an evaluation, and hand it back to the LLM to retry.

5.4 Data Pre-selection

As I mentioned earlier, once a large number of scenarios are onboarded, you will inevitably hit the Token limit of a single LLM pass, and long text also weakens the LLM’s information extraction ability.

Pre-selecting data avoids these problems by narrowing the scope to information relevant to the input.

As shown above, there may be many types of data to pre-select, for example:

• Execution templates: Pipeline and Task templates
• Context: domain information relevant to the current input; for operations that means Metrics, Logs, Events, and so on — though these can also be queried in real time through Tasks
• Team personnel information, such as the owner of an application
• Permission information: when executing a Permission Task, the user’s permissions need to be validated
• Knowledge base: here you need a vector database for recall
• Some examples and successful Cases, for the LLM to do few-shot learning

6. Summary

This post is mainly a summary of what I have learned recently while developing AI Agent applications. The main contents are as follows:

1. When landing AI Agent applications within a team, finding an iterable technical path is important — it helps with reporting and showing staged results.

2. I introduced a design approach for a service-oriented AI Agent that splits it into an AI side responsible for analysis and an Agent side responsible for execution, the two interacting through APIs. A service-oriented AI Agent is good for integrating existing systems, and avoids having an AI Agent integrate every system.

3. I introduced an iteration approach for developing a service-oriented AI Agent: the AI side uses a Pipeline object and the Agent side uses a Task object. First, align with specific scenarios through built-in Pipelines; then improve precision at large scale through vector retrieval; then let the AI Agent wake itself to onboard the full set of scenarios; and finally let the LLM compose Pipelines autonomously to complete tasks.

4. I introduced several methods for solving alignment problems when developing AI Agent applications, including accurate function and field descriptions, a Retry mechanism, a Reflection\Reward mechanism, and data pre-selection.

What is learned from paper is after all superficial; only by practicing it yourself do you truly understand the matter. Only when I actually sat down to write an AI Agent to solve real problems did I discover how many pitfalls there are. I record it here in this post.