1. What Is an Ops Tool

https://www.chenshaowen.com/ops/ is one of the tools I use most often in day-to-day operations.

To operate on machines, I can reuse my previous scripts and perform operations in bulk.

To operate on clusters, I can reuse my previous scripts and operate machines without logging into the nodes.

If I run into a new operations problem, I will immediately write a Task Yaml to codify the operation, so it can be reused next time.

The core operations of Ops are script execution and file distribution, and its core objects are hosts and Kubernetes clusters. Both hosts and clusters need to implement Ops’ two core operations, and on top of that is Task orchestration, which captures operations scenarios.

2. Why Write a Copilot

To try developing applications based on LLMs. LLMs provide an entirely new way of using and designing things, and represent an advanced mode of production. For humans, repetitive, tedious, and mechanical work can all be handed to an LLM as long as enough data has been accumulated. Applications will evolve in the direction of being based on LLMs, and I also want to think more about the possibilities for putting this direction into practice.

To make up for the insufficient scenarios covered by the Ops tool. Ops can deliver a limited set of core operations capabilities and connect to a limited set of infrastructure. But it cannot satisfy an unlimited range of operations scenarios. Open-ended scenarios need an agent like an LLM to break the scenario down and convert it into Ops’ already-implemented core capabilities executed according to a certain logic. This conversion from open-ended scenarios to a limited set of core capabilities is a problem that applications based on LLMs need to consider carefully.

Writing a Copilot is simpler than I imagined, but it needs a usage scenario to drive thinking and iteration. A while ago I had been busy with LLM inference, and this week I happened to have time to work on Ops. From having the idea, to starting to write it, to getting some results, it took less than half a week, and the actual time spent writing code was less than a day.

It can improve my work efficiency. Ops currently has three components: Opscli, OpsServer, and OpsController. Opscli is a command-line tool, very similar in form to an LLM front end, and it is also a component I use frequently. Integrating a Copilot into Opscli helps save me the time needed to troubleshoot failures and change configurations.

3. Let Me Talk About My Approach

3.1 Processing Flow

As shown above:

1. The user’s input may be a piece of text, a click, or of course an event
2. The LLM needs to take this input and convert it into a number of steps inside the system
3. My application executes this series of tasks
4. The LLM organizes the execution results of these tasks and produces an output. This output may be a hint, a popup, a piece of text, or an event.

3.2 Breaking Down Tasks

In the processing flow above, the first difficulty is how to break down the task.

As shown above:

Human thinking is directional, and human descriptions are abstract.

For example, when a person is hungry they say, “I’m hungry,” “I want to eat.” But this kind of expression is incomprehensible to a machine. You have to say, “Half an hour from now, at home, I will eat, one bowl of rice, one tomato scrambled eggs, one stir-fried greens” — only then is it something a machine can understand and execute.

This process of converting an abstract task into a concrete task TodoList is one of the core functions the Copilot needs to accomplish.

In the process of breaking down tasks, besides the intelligence of the LLM itself, we also need:

• Broad domain knowledge. What an LLM has is general knowledge, such as cultural habits, language habits, and behavioral habits, but for specific, up-to-date domain knowledge, the LLM has not had time to learn it, and there is not enough corpus to learn from either. If domain knowledge can be loaded through dynamically loaded lora fine-tuning or a vector knowledge base, that would be a very good approach.
• Information related to my application. To the LLM, my application is a black box — a brand-new, unfamiliar, unknown thing. Only by letting the LLM understand my application can it break down tasks better.
• Context. Every response to an event needs to consider context. Which user, which database, which cluster, which operating system, and so on — these make up the context information.

With these supplements, the LLM can convert abstract tasks into concrete tasks for my application to execute.

3.3 Exposing Application Information to the LLM

This is another place where I need to keep optimizing the Ops tool.

As shown above:

To let the LLM better understand my application, I need to submit detailed and complete documentation of the application to the LLM. But submitting all the documentation to the LLM is not feasible. You not only have to overcome the maximum text length the LLM can process, but also consider the problem of information loss in the middle section caused by long text. Therefore, the best approach is to submit a Schema — submit your design approach, not your design implementation.

The same applies to the implementation of the API\CLI: do not submit every single interface URL, parameter, and return value to the LLM; instead, tell the LLM what your API\CLI design approach is.

But this places very high demands on the application’s designer. We can no longer follow the crowd and copy from everywhere; instead, we need the product to have a simple, self-consistent, unified design.

4. Current Progress and Future Plans

At present, the Ops Copilot is still at a very early stage, having implemented functionality similar to open-interpreter.

opscli copilot can, based on the user’s input, execute scripts locally and output results, completing tasks such as opening a browser, querying various information about a Kubernetes cluster, and closing the WeChat application.

If development goes smoothly, I hope opscli copilot can evolve into a separate service, OpsCopilot, connecting to OpsCli, OpsServer, and OpsController.

At the same time, Copilot will not depend on features specific to models such as GPT-3.5 or GPT-4, such as function call, keeping Copilot general-purpose rather than compatible.