This page looks best with JavaScript enabled

You Can Also Land AI Agents This Way - The Operations Events Edition

 ·  ☕ 6 min read

1. Why Operations Events

  • Less invasive

Operations teams usually already have observations accumulated in the Metrics and Log dimensions. Operations events can be a new entry point: introduce new operational capabilities without affecting the stability of existing systems.

  • Better suited to automation

Metrics represent system state, Log represents specific code behavior, while Event represents changes in components.

And change is the greatest enemy of stability. Operations events capture change better and can trigger automation.

  • Faster response

The anomaly detection mechanism for Metrics and Log is scheduled queries, along with configured threshold and duration checks, so alerting carries a certain delay.

The anomaly detection mechanism for operations events is real-time listening: as soon as an event is watched, automation can be triggered immediately.

  • I can offer some experience

Over the past two years I have been exploring the application of AI Agents in operations, and there are already some cases to reference. We generate 1M+ operations events every day; through various alerts and fully automated handling flows, they keep dozens of AI clusters and the AI applications on them running stably.

2. Installing the Event Components

  • Install Nats
1
2
3

helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm repo update
helm show values nats/nats
  • Generate nats-values.yaml
1
2

export adminpassword=mypassword
export apppassword=mypassword
  • Generate nats-values.yaml

Note the storageClassName needs adjusting

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

cat <<EOF > nats-values.yaml
config:
  jetstream:
    enabled: true
    fileStore:
      enabled: false
      dir: /data
    memoryStore:
      enabled: true
      maxSize: 1Gi
    pvc:
      enabled: false
      storageClassName: my-sc
  cluster:
    enabled: true
  leafnodes:
    enabled: true
  merge:
    accounts:
      SYS:
        users:
          - user: admin
            password: ${adminpassword}
      APP:
        users:
          - user: app
            password: ${apppassword}
        jetstream: true
    system_account: SYS
container:
  image:
    repository: nats
    tag: 2.10.20-alpine
natsBox:
  container:
    image:
      repository: nats-box
      tag: 0.14.5
reloader:
  enabled: true
  image:
    repository: natsio/nats-server-config-reloader
    tag: 0.15.1
EOF

For multiple clusters, you can refer to https://www.chenshaowen.com/ops/zh/nats.html to configure a primary Nats cluster, with the other clusters connecting to it through leafnodes.

  • Install Ops
1
2
3
4
5
6
7
8

helm install myops ops/ops
            --version 1.2.0 \
            --namespace ops-system \
            --create-namespace \
            --set controller.env.activeNamespace="ops-system" \
            --set controller.env.defaultRuntimeImage="ubuntu:22.04" \
            --set event.cluster="mycluster" \
            --set event.endpoint="http://app:mypassword@nats-headless.ops-system.svc:4222"

The https://github.com/shaowenchen/ops project watches all Kubernetes events, converts them into operations events, and stores them in an external Nats cluster. Confirmed through metric monitoring, the Ops project does not affect the performance of kube-apiserver or etcd. Once events are exported and stored in Nats, they put no additional pressure on Kubernetes.

3. Event Format

3.1 Namespace-Level Event Format

1

ops.clusters.{cluster}.namespaces.{namespace}.{resourceType}.{resourceName}.{observation}

Among them:

  • {cluster}: cluster name (read from the environment variable EVENT_CLUSTER)
  • {namespace}: Kubernetes namespace
  • {resourceType}: resource type (such as deployments, pods, configmaps, etc.)
  • {resourceName}: resource name
  • {observation}: observation type, with the possible values and their meanings:
    • status: resource status information (such as running state, health state, etc.)
    • events: event information (such as Kubernetes events)
    • alerts: alert information
    • findings: proactively reported information and state

3.2 Node Event Format

1

ops.clusters.{cluster}.nodes.{nodeName}.{observation}

Among them:

  • {cluster}: cluster name (read from the environment variable EVENT_CLUSTER)
  • {nodeName}: node name
  • {observation}: observation type, with the possible values and their meanings:
    • events: event information (such as Kubernetes events)
    • alerts: alert information
    • findings: proactively reported information and state

Example:

1

ops.clusters.mycluster.nodes.mynode.events

3.3 Notification Format

This kind of event data is not collected from Kubernetes by the Ops project; instead it is used to gather various notifications from other systems and turn them into operations events that can be handled.

Notification event subject format:

1

ops.notifications.providers.{provider}.channels.{channel}.severities.{severity}
  • {provider}: notification provider or system name (such as ksyun, ai, etc.)
  • {channel}: notification channel type (such as webhook, email, sms, etc.)
  • {severity}: severity level (such as info, warning, error, critical, etc.)

Example:

1

ops.notifications.providers.ksyun.channels.webhook.severities.critical

4. Handling Events

4.1 Triggering Notifications Directly

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: crd.chenshaowen.com/v1
kind: EventHooks
metadata:
  name: nodes-p0
  namespace: ops-system
spec:
  keywords:
    exclude:
    - cn-beijing
    include:
    - NodeNotReady
    - NodeReady
  subject: ops.clusters.*.nodes.*.events
  type: xiezuo
  url: https://x.x.x/api/v1/webhook/send?key=

When a cluster node is not ready, a notification goes out immediately, minutes faster than a Metrics alert.

4.2 Triggering Automation Flows

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

apiVersion: crd.chenshaowen.com/v1
kind: EventHooks
metadata:
  name: custom-disable
  namespace: ops-system
spec:
  additional: 'action: cordon-node'
  keywords:
    include:
    - "10005"
  subject: ops.clusters.*.nodes.*.events
  type: webhook
  url: http://x.x.x.x:x/trigger/action?key=

When “10005” is detected in a node event, an automation flow is triggered to perform the “cordon-node” action.

4.3 Converting Events

To make it convenient to consume events with a given characteristic, we first convert events into alert events. The exception handling module then only needs to listen for alert events.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: crd.chenshaowen.com/v1
kind: EventHooks
metadata:
  name: convert-clusters-pods-events-to-alerts
  namespace: ops-system
spec:
  keywords:
    include:
      - .*(BackOff|OOMKilled|Evicted|NetworkNotReady|Unhealthy|Error|Failed|ImagePullBackOff).*
    matchMode: ALL
    matchType: REGEX
  subject: ops.clusters.*.namespaces.*.pods.*.events
  type: event
  url: ops.clusters.*.namespaces.*.pods.*.alerts

When BackOff is detected in ops.clusters.mycluster.namespaces.mynamespace.pods.myapp.events, an ops.clusters.mycluster.namespaces.mynamespace.pods.myapp.alerts event with the same content is created.

4.4 Analyzing Events

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

apiVersion: crd.chenshaowen.com/v1
kind: EventHooks
metadata:
  name: post-cluster-pod-ops-agent
  namespace: ops-system
spec:
  additional: clusterpodevents
  keywords:
    include:
    - .*(kube-apiserver-|kube-controller-manager-|kube-scheduler-|kube-proxy-|etcd-|calico-|csi-nfs-|fluid-system).*
    matchMode: ANY
    matchType: REGEX
  subject: ops.clusters.*.namespaces.*.pods.*.alerts
  type: webhook
  url: http://x.x.x.x:x/api/push

All you need to do is develop an /api/push endpoint that receives operations events and analyzes and handles them.

In our system, this endpoint already aggregates operations events from the three dimensions of applications, nodes, and clusters, for systematic analysis.

5. How to Ingest Events

5.1 Ingesting Alert Events

The https://github.com/shaowenchen/ops project provides an ops-server service that can push and consume operations events.

1

http://myops-server.ops-system.svc:80/api/v1/namespaces/ops-system/events/ops.notifications.providers.aliyun.channels.webhook.severities.info

1

http://myops-server.ops-system.svc:80/api/v1/namespaces/ops-system/events/ops.notifications.providers.ksyun.channels.webhook.severities.info

These are the event storage Topics for two cloud providers. Beyond that I have also connected an internal alert notification system, gradually improving the system event observation capability.

5.2 taskrun Scheduled Jobs

The https://github.com/shaowenchen/ops project supports scheduled jobs; you can configure taskrun to run some customized script checks and report them as findings events.

1
2
3
4
5
6
7
8

apiVersion: crd.chenshaowen.com/v1
kind: TaskRun
metadata:
  name: alert-hosts-card
  namespace: ops-system
spec:
  crontab: 30 * * * *
  taskRef: alert-hosts-card

In ops.clusters.*.namespaces.*.hosts.*.findings, these events will be received.

The Ops project also provides a web page for viewing these events.

6. Summary

In 2025 I did a series of optimizations in the AI Agent operations direction: stopping iteration on the previous copilot mode, introducing MCP to unify the ingestion of Metrics, Log, Event, and SOPS, structuring alert data, triggering SOPS directly, and automating SLO analysis.

The cornerstone of these automation-layer optimizations is operations events. I previously wrote Transforming the Operations System with an Event Bus explaining the important role of the event bus in an operations system, and this post is a real-world implementation of that event bus.

微信公众号
WRITTEN BY
微信公众号

See Also