1. What Is ChatOps

GitOps, ChatOps, AIOps, and the like (hereafter NewOps) are emerging operations philosophies that have appeared in recent years. NewOps separates Ops out of its chaotic state into two parts: one facing users, trending toward being more human, auditable, and traceable; the other facing infrastructure, trending toward being more programmatic, automated, and intelligent.

Usually, when we talk about NewOps, we emphasize the human-collaboration part and overlook the underlying systems. Think about it: without the support of IaC tools, could GitOps really work? Without a powerful AI system, how would you do AIOps? Git and Chat serve only as an interactive front end; the key advances in NewOps come from the evolution of the underlying Ops.

Below, let’s focus on ChatOps.

As shown above, ChatOps is driven from the front end by a chat tool, while on the back end a bot performs operations and updates the infrastructure. Here is a brief introduction to two ChatOps tools on GitHub.

• Prow

In the document Prow of the DevOps Tool Chain, I shared this before. It drives the development process through labels, is widely practiced in the Kubernetes community, and in my current team I am mainly the one maintaining it.

• Hubot

Hubot is a chat bot open-sourced by GitHub, configured using .coffee or .js files, and it can execute actions from comments. Here is an official example:

1
2
3
4
5
6
7
8
9

module.exports = (robot) ->
  robot.hear /badger/i, (res) ->
    res.send "Badgers? BADGERS? WE DON'T NEED NO STINKIN BADGERS"

  robot.respond /open the pod bay doors/i, (res) ->
    res.reply "I'm afraid I can't let you do that."

  robot.hear /I like pie/i, (res) ->
    res.emote "makes a freshly baked pie"

Of course there are other tools or approaches, such as Slack Bot — these are just examples. Think about ChatOps again: it is really a matching system that matches keywords and then executes the corresponding actions.

It is that simple, so the bot here does not actually have to be an external service; for small needs, a few lines of script can get the job done.

2. Team Needs and Results

Here is a brief look at the team’s development model.

• Front-end and back-end separation
• For a new feature, the back end finishes the API development first, then the back end deploys the service to a designated environment, and the front end calls the API in that environment to develop the new feature

The repository uses a Monorepo and is preparing to move to Polyrepo. Right now all components are coupled together in one repository; in the future they will be split into several separate repositories.

The current problem with the Monorepo is that testing has not kept up, so Pull Requests cannot be merged quickly, versions cannot iterate rapidly, and the speed of agile development drops. Another technique here is feature flags, but feature flags in turn mean adding redundant, compatibility code.

To minimize the impact on developers, the main thing is still to make testing the entry point. In the earlier document Using Terraform and GitHub Actions to Automate Infrastructure Installation Testing, I ran automated tests at the delivery convergence point of all components. Here, the main thing is to provide a preview environment for each Pull Request.

The front end submits a PR; after waiting for All checks to complete, you only need to reply /deploy to get a preview link.

After preview verification is complete, you only need to reply /clear to clean up the load generated by the preview.

On the first day it went live, the person responsible for the front-end repository found a problem in a PR ahead of time through the preview — the results were pretty good.

3. References

• https://hubot.github.com/docs/scripting/