Aligning with Ops: Rewriting Ops Copilot with a New Approach, Updated

1. Making Ops Copilot an Ops Coilot

In September 2023, I wrote a version of Ops Copilot, and published an article about it, I’m Writing a Copilot for My Ops Tool.

The effect it achieved was like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
Opscli> 打开浏览器
Open a browser and navigate to 'https://www.google.com'.
↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
import webbrowser

webbrowser.open('https://www.google.com')
↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑

Can I run this code? (y/n)
y

At that point, the local default browser would open.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
Opscli> 获取 K8s 节点信息
Retrieve information about Kubernetes nodes
↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓↓
kubectl get nodes
↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑

Can I run this code? (y/n)
> y
NAME    STATUS   ROLES                         AGE    VERSION
node1   Ready    control-plane,master,worker   407d   v1.21.0

But for the Ops project, this kind of feature was meaningless — plenty of open-source projects can provide the same capability.

During the interactive conversation, Ops Copilot completely ignored the capabilities Ops provides. That was the crux of the problem, and I paused updating Copilot-related features for a while.

Ops Copilot’s most recent iteration came out of a production need at my job: resolving various AI infrastructure incidents. The idea came from service-oriented AI Agents and pipelined AI tasks.

Let’s take a look at several key points of this iteration.

2. The Design of Ops Copilot

Ops Copilot is a CLI subcommand that uses an LLM to integrate Ops capabilities. It mainly aims to solve the following problems:

How to choose the right Task to solve a problem
How to extract the parameters for executing the relevant Task

Below is the processing logic of Ops Copilot:

Scenarios are described through Pipeline definitions, and with the help of an LLM the text input is converted into the execution of a particular Pipeline, with the relevant parameters extracted.

Ops Copilot parses the Pipeline and executes the Task jobs. Ops Copilot interacts with Ops Controller through Ops Server, creates a TaskRun job, waits for Ops Controller to finish executing, and retrieves the result.

Ops Server is the component added when Ops gained a new UI iteration in early 2024. It provides authentication and can create Task execution instances, TaskRun.

Ops Controller watches for TaskRun creation, and based on the Cluster or Host object specified in it, obtains the relevant credentials to connect to the Ops Object and perform the relevant operations.

And operations on Cluster and Hosts objects — script execution and file distribution — are precisely the core design of the Ops project.

Following this approach, Ops Copilot is a subcommand truly built on the Ops project, integrated into it, as shown below.

3. Using Ops Copilot

3.1 Parameter Reference

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
/usr/local/bin/opscli copilot --help

use llm to assist ops

Usage:
  opscli copilot [flags]

Flags:
  -e, --endpoint string   e.g. https://api.openai.com/v1
  -h, --help              help for copilot
      --history int        (default 5)
  -k, --key string        e.g. sk-xxx
  -m, --model string      e.g. gpt-3.5-turbo
  -s, --silence
  -v, --verbose string

By default, copilot reads OPENAI_API_HOST, OPENAI_API_BASE, OPENAI_API_MODEL, OPENAI_API_KEY, OPS_SERVER, and OPS_TOKEN from environment variables; if an environment variable is not present, the default value is used.

3.2 Installation

If you already installed it before, run opscli upgrade to upgrade to the latest version.

For use in China

1
curl -sfL https://raw.githubusercontent.com/shaowenchen/ops/main/getcli.sh |VERSION=latest sh -

For use outside China

1
curl -sfL https://raw.githubusercontent.com/shaowenchen/ops/main/getcli.sh | VERSION=latest sh -

3.3 Usage

Set the environment variables

1
2
3
4
export OPENAI_API_KEY=sk-xxxx
export OPENAI_API_HOST=https://llmapi.YOUR-OPENAI-SERVER.com/v1
export OPS_SERVER=http://1.1.1.1
export OPS_TOKEN=xxxx

Run Copilot

1
2
3
4
/usr/local/bin/opscli copilot

Welcome to Opscli Copilot. Please type "exit" or "q" to quit.
Opscli>

See which operations are supported

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
Opscli> 有哪些可用的操作
这里列出了可用的操作及其相应的描述和变量：

1. list-cluster：查询K8s集群列表。
2. list-task：查询任务列表。
3. list-pipeline：查询流水线列表。
4. restart-pod：重新启动或删除Pod。变量：podname（一个或多个Pod名称）。
5. force-restart-pod：强制重新启动或删除Pod。变量：podname（一个或多个Pod名称）。
6. get-cluster-ip：查询集群的IP地址。变量：clusterip（一个或多个集群IP地址）。
7. clear-disk：清空磁盘。变量：nodeName（一个或多个节点名称）。

您可以根据具体需求选择合适的操作来执行相应的操作。

These pipelines are defined in https://github.com/shaowenchen/ops/blob/main/pkg/copilot/pipelines.go. I hope there will be a better extension mechanism later, for example auto-discovery from the clusters where Ops is deployed.

Query which clusters exist

1
2
3
4
5
6
Opscli> 有哪些集群
这些集群如下：
1. ops-system/xx-xx：该集群部署在xxx云上的推理集群
2. ops-system/xx-xx：该集群部署在xxx上的集群
3. ops-system/xx-xx：该集群部署在xxx上的训推一体集群
4. ops-system/xx-xx：该集群部署在xx上的 NPU 训练集群

Ops Copilot periodically pulls the cluster list and host list data from the clusters where Ops is deployed.

Restart a Pod

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
Opscli> 强制重启训推一体集群上的pod ubuntu-8474647969-qszcj
强制重启训推一体集群上的pod ubuntu-8474647969-qszcj

- 步骤：检查pod是否存在
- 输出：
在默认命名空间中找到了Pod ubuntu-8474647969-qszcj。

- 步骤：删除pod
- 输出：
警告：立即删除不会等待确认正在运行的资源是否已终止。该资源可能会无限期地在集群上运行。

Pod "ubuntu-8474647969-qszcj" 已被强制删除。

On the cluster you can see the relevant Pod’s status update.

1
2
3
4
5
kubectl get pod ubuntu-8474647969-qszcj -w
NAME                      READY   STATUS    RESTARTS   AGE
ubuntu-8474647969-qszcj   1/1     Running   0          20h
ubuntu-8474647969-qszcj   1/1     Terminating   0          20h
ubuntu-8474647969-qszcj   1/1     Terminating   0          20h

A pipeline usually contains several Task jobs. These Tasks come from two sources:

Task jobs that Ops Copilot periodically obtains from the clusters where Ops is deployed; there are many of these in the Ops Github repository
Hard-coded Task jobs, https://github.com/shaowenchen/ops/blob/main/pkg/copilot/tasks.go, which allow extending Task jobs unrelated to Ops, such as summary, which uses an LLM to summarize the output, and allow-whitelist, which controls the operator whitelist, etc.

4. Alignment Is the Key

4.1 Aligning on Scenario Intent

Because of my professional experience — I have rolled out Gitlab CI, developed a Github Action plugin, iterated a Jenkins-based project, and built projects based on Tekton and Argo Workflow from 0 to 1 — when I faced the need to design execution logic, my instinct was to choose the approach of assembling tasks into pipelines to execute AI tasks.

Below is a pipeline definition:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
var pipelineListPipeline = agent.LLMPipeline{
	Desc:      "Query -  list pipeline",
	Namespace: "ops-system",
	Name:      "list-pipeline",
	NodeName:  "anymaster",
	LLMTasks: []agent.LLMTask{
		{
			Name: "list-pipelines",
		},
		{
			Name: "summary",
		},
	},
}

Desc tells the LLM what this Pipeline is for
LLMTasks contains the concrete logic the Pipeline executes

There are also some default values and configuration, all of which are there to let the LLM match the input accurately while constraining the LLM within the rules we defined.

Desc accurately provides each pipeline’s purpose to the LLM, and the LLM chooses which Pipeline to execute, based on the input text, to solve the problem described by the text — while what Task that Pipeline actually executes is something we work out.

4.2 Aligning on Variable Extraction

Variable extraction is another key point. See the example below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
var pipelineRestartPod = agent.LLMPipeline{
	Desc:      "Restart - Restart、delete Pod",
	Namespace: "ops-system",
	Name:      "restart-pod",
	NodeName:  "anymaster",
	Variables: []agent.VariablePair{
		{
			Key:      "podname",
			Desc:     "For example, `pod: long-v1-64cf8d5478-5zsvk or name: long-v1-64cf8d5478-5zsvk`, where long-v1-64cf8d5478-5zsvk is podname",
			Required: true,
		},
	},
	LLMTasks: []agent.LLMTask{
		{
			Name: "check-pod-existed",
		},
		{
			Name: "delete-pod",
		},
		{
			Name: "summary",
		},
	},
}

There are two ways to extract variables effectively:

Use regex matching
Give examples

In the example above, providing a real example of podname helps the LLM extract the variables the pipeline needs from the output text.

5. Event-Driven AI Agents

In an earlier article I mentioned that Copilot is driven by people and is people’s responsibility, while an AI Agent is driven by the program itself and is the program’s responsibility.

Through the alignment approaches above, this is in fact not limited to the operations domain; in other domains you can also rapidly align an LLM to implement Copilot functionality based on pipelined AI tasks. The hard part is that you need to implement domain abstractions and component capabilities in that domain similar to Ops Server and Ops Controller. That is not an easy thing.

The original opportunity for developing Ops came from a work need: I had to maintain many machines and clusters, doing operations and configuration in bulk. Thanks to the team’s tolerance, I could develop the Ops project while trying out a large number of operations tasks, and eventually Ops gradually stabilized and came to be used in production. During the iterations, a bug in the Ops project nearly caused an incident; I recorded that part in my blog too, and afterward I treated Ops updates with increasing caution.

Back to AI Agents: there is now also a production need at my job — automatically handling online AI infrastructure failures.

In fact, Copilot and an AI Agent are separated by only a thin wall: it is nothing more than turning passive into active, using a program instead of a person to drive Copilot.

As shown above, Copilot is triggered based on a person’s output, analyzes the person’s output, executes tasks, and feeds the result back to the person. But where does the person’s output come from? There are mainly two kinds:

From 0 to 1, the original requester of the need

The part created autonomously and originally — for example, I want to create a namespace. Then where will this idea be recorded, and can the source of that record serve as an event source that automatically triggers Copilot to execute the task of creating a namespace.

From 1 to 1, copied from elsewhere

The part copied from elsewhere — for example, an alert message. Can it generate an event, notify a person, and after confirmation and authorization, be handed to Copilot for execution.

Every activity around production can be turned into an event and passed to Copilot — that is the event-driven AI Agent development approach.

Of course, I am also in the early stages of practicing this part, because inspections can also be done through Ops.

As shown above, inspection tasks run periodically, and when an anomaly is found an event is generated and a notification is sent. If this event were sent directly to Copilot, and Copilot had already integrated a way to handle this kind of event, then the anomaly could be resolved directly with the help of an LLM, achieving full takeover of failure handling.

6. Summary

The main content of this post is as follows:

The redesign of Ops Copilot

Ops Copilot was redesigned so that it integrates more deeply with the Ops project, using large language models (LLMs) to enhance the automation of operations tasks.

Pipelined AI tasks

By pipelining AI tasks, and drawing on experience from different CI/CD tools and workflow projects, task assembly and execution were implemented in Ops Copilot.

Event-driven AI Agents

Turning human operations into program-driven events can promote Copilot’s evolution toward an AI Agent and achieve a program’s full takeover of the environment.