Using Falco to Monitor Runtime Security

1. What Falco Is

Falco is a cloud-native runtime security project contributed to the CNCF by Sysdig.

Falco implements an extensible event rule filtering engine. Through a series of operations — capturing events, matching security rules, and producing alert notifications — it can surface security problems in a system. The events come from system calls, and it also supports an ebpf probe; the rules are open source [1] and you can define your own extensions [2]. The architecture diagram is below:

Typical events Falco can detect include:

A shell running inside a container
A container running in privileged mode
Reading sensitive data, such as /etc/shadow
A container mounting a sensitive host path
Outbound network connections

2. Generating Certificates

Falco’s gRPC requires mutual TLS authentication [3]. The Falco exporter exposes the relevant events over gRPC, and a system you build yourself can also integrate directly with Falco over gRPC. The steps below generate the certificates needed for the interaction; Falco’s official documentation is a bit dated and some of the operations will fail.

Create the certificate directory

1
2
mkdir /root/falco
cd /root/falco

Create the CA certificate

1
2
3
4
5
6
openssl genrsa -out ca.key 4096

openssl req -x509 -new -nodes -sha512 -days 3650 \
        -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=dev.chenshaowen.com" \
        -key ca.key \
        -out ca.crt

Create the server certificate

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
openssl genrsa -out server.key 4096

openssl req -sha512 -new                  \
              -key server.key       \
              -out server.csr       \
              -subj  "/C=SP/ST=Italy/L=Ornavasso/O=Test/OU=Server/CN=localhost"

openssl x509 -req -sha512               \
               -days 3650            \
               -CA ca.crt           \
               -CAkey ca.key        \
               -in server.csr       \
               -out server.crt      \
               -set_serial 01

Create the client certificate

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
openssl genrsa -out client.key 4096

openssl req -sha512 -new                  \
              -key client.key       \
              -out client.csr       \
              -subj  "/C=SP/ST=Italy/L=Ornavasso/O=Test/OU=client/CN=localhost"

openssl x509 -req -sha512               \
               -days 3650            \
               -CA ca.crt           \
               -CAkey ca.key        \
               -in client.csr       \
               -out client.crt      \
               -set_serial 01

List all the generated certificates

1
2
3
ls /root/falco

ca.crt  ca.key  client.crt  client.csr  client.key  server.crt  server.csr  server.key

3. Installing Falco on Kubernetes

Add the Helm repository

1
2
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Install falco [4]

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
helm install falco falcosecurity/falco \
      --namespace falco --create-namespace \
      --version 3.0.0 \
      --set-file certs.ca.crt=/root/falco/ca.crt,certs.server.key=/root/falco/server.key,certs.server.crt=/root/falco/server.crt \
      --set ebpf.enabled=true \
      --set falco.grpc.enabled=true \
      --set falco.grpc_output.enabled=true \
      --set falcosidekick.enabled=true \
      --set falcosidekick.webui.enabled=true \
      --set falcosidekick.webui.user="admin:admin" \
      --set falco.grpc.unixSocketPath=""

Install falco-exporter

1
2
3
4
5
helm install falco-exporter falcosecurity/falco-exporter \
     --namespace falco --create-namespace \
     --version 0.9.1 \
     --set falco.grpcTimeout=3m
     --set-file certs.ca.crt=/root/falco/ca.crt,certs.client.key=/root/falco/client.key,certs.client.crt=/root/falco/client.crt

Check the services

During startup it requests ghcr.io to download the default rules, falco_rules.yaml.tar.gz. In a network-restricted environment, this download may fail.

1
2
3
4
5
6
7
8
kubectl -n falco get pod -w

NAME                                      READY   STATUS    RESTARTS   AGE
falco-5bbl6                               2/2     Running   0          152s
falco-exporter-26gfz                      1/1     Running   0          124s
falco-falcosidekick-5c8bf5d7fb-kx778      1/1     Running   0          111s
falco-falcosidekick-ui-5b56bbd7cb-5wdwl   1/1     Running   3          111s
...

[Optional] Uninstalling Falco

1
2
helm uninstall falco --namespace falco
helm uninstall falco-exporter --namespace falco

4. Viewing Falco Event Data with a Grafana Dashboard

Looking at the falco-exporter svc, you can see that it has already exposed its metrics to Prometheus.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
kubectl -n falco get svc falco-exporter  -o yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9376"
    prometheus.io/scrape: "true"
  name: falco-exporter
  namespace: falco
spec:
  ports:
  - name: metrics
    port: 9376
    protocol: TCP
    targetPort: 9376
  selector:
    app.kubernetes.io/instance: falco-exporter
    app.kubernetes.io/name: falco-exporter

Next, all you need to do is add a Grafana dashboard. Import 11914, which is https://grafana.com/grafana/dashboards/11914-falco-dashboard/. The data looks like the figure below:

But the information exposed in the metrics is fairly limited. Querying falco_events{rule="Read sensitive file untrusted"} in Prometheus gives the result falco_events{app_kubernetes_io_instance="falco-exporter", app_kubernetes_io_managed_by="Helm", app_kubernetes_io_name="falco-exporter", app_kubernetes_io_version="0.8.0", helm_sh_chart="falco-exporter-0.9.1", hostname="falco-h57xg", instance="1.1.1.1:9376", job="kubernetes-service-endpoints", k8s_ns_name="<NA>", k8s_pod_name="<NA>", namespace="falco", node="node1", priority="4", rule="Read sensitive file untrusted", service="falco-exporter", source="syscall", tags=",T1020,T1083,T1212,T1552,T1555,container,filesystem,host,mitre_credential_access,mitre_discovery,"} , and it does not show details such as the executing user or the command that ran — you can only see limited information such as the event’s priority and the rule that triggered.

5. Viewing Events with falcosidekick-ui

falcosidekick mainly provides centralized event management and rich alerting channel capabilities, able to send alerts to slack, rocketchat, elasticsearch, and others [5].

falco-falcosidekick-ui, in turn, provides the ability to view falco events.

Install falco-falcosidekick-ui

When installing falco just now, the following parameters were already added, so falcosidekick and falcosidekick-ui are already installed.

1
2
3
      --set falcosidekick.enabled=true \
      --set falcosidekick.webui.enabled=true \
      --set falcosidekick.webui.user="admin:admin"

Expose the service port

1
2
kubectl -n falco patch svc falco-falcosidekick-ui  --patch \
  '{"spec": { "type": "NodePort", "ports": [ { "nodePort": 32000, "port": 2802, "protocol": "TCP", "targetPort": 2802 } ] } }'

Through the host IP at port 32000, you can open the falcosidekick ui page. The default account is admin and the default password is admin.

On the host, after reading a sensitive file with cat /etc/shadow, you can see the related event on the falcosidekick ui page, as shown below:

The Output content, since the operation is on the host, will be missing the cluster-related fields, but the file, command, and other information is considerably richer:

1
Warning Sensitive file opened for reading by non-trusted program (user=root user_loginuid=1001 program=cat command=cat /etc/shadow pid=54909 file=/etc/shadow parent=bash gparent=sudo ggparent=bash gggparent=sshd container_id=host image=<NA>) k8s.ns=<NA> k8s.pod=<NA> container=host

The Dashboard page provides a global statistics view, which lets you assess the security of clusters and hosts as a whole. The view is as follows: