EnvoyFilter Configuration in Istio

1. What Is EnvoyFilter

EnvoyFilter is a CRD resource in Istio that allows users to modify Envoy’s configuration to satisfy their customization needs for different scenarios.

1
2
3
4
5
6
7
8
9
kubectl get envoyfilter -A

NAMESPACE            NAME                                              AGE
istio-system         add-request-id-into-ingressgateway                54d
istio-system         compression-gzip                                  18d
istio-system         custom-access-log                                 3d
istio-system         ingressgateway-settings                           52d
istio-system         preserve-request-header-us-test-ingress-gateway   95d
istio-system         preserve-x-request-id                             54d

When using Istio, you will usually end up using at least a few EnvoyFilters.

The features EnvoyFilter provides are built on Envoy’s existing built-in functionality and extension mechanisms, and mainly include:

Modify network requests and responses
Rate limiting and circuit breaking
Retries
Modify routing rules
Add extra monitoring metrics
Access allowlist/denylist control
Run Lua scripts
…

It covers most Layer 4 and Layer 7 requirements. If it cannot, you can move up to WasmPlugin; I previously wrote Developing an Istio WasmPlugin with tinygo
for reference.

2. EnvoyFilter Caveats

An incorrect configuration can undermine the stability of the entire mesh, so EnvoyFilter must be used with great care.

The scope in which an EnvoyFilter takes effect:

Global: an EnvoyFilter in the root namespace. The root namespace is defined in the configuration file, default rootNamespace: istio-system
Named namespace: an EnvoyFilter created in a specific namespace

EnvoyFilter priority:

Configuration in the root namespace > configuration in other namespaces

EnvoyFilter application order:

Applied in order of creation time

3. EnvoyFilter Configuration Fields

Let’s start with an example: Gzip-compressing the Response that passes through the Istio Gateway.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: compression-gzip
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: istio-ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.compressor
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.compressor.v3.Compressor
            response_direction_config:
              common_config:
                min_content_length: 256
                content_type:
                  - application/atom+xml
                  - application/javascript
                  - application/x-javascript
                  - application/json
                  - application/rss+xml
                  - application/vnd.ms-fontobject
                  - application/x-font-ttf
                  - application/x-web-app-manifest+json
                  - application/xhtml+xml
                  - application/xml
                  - font/opentype
                  - image/svg+xml
                  - image/x-icon
                  - text/css
                  - text/javascript
                  - text/plain
                  - text/x-component
            compressor_library:
              name: text_optimized
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.compression.gzip.compressor.v3.Gzip
                memory_level: 3
                compression_level: COMPRESSION_LEVEL_6

A brief note on the key fields

applyTo

The filter layer it applies to; here it points to the HTTP filter layer

workloadSelector

Specifies the Pods it takes effect on; if workloadSelector is not set, it applies to the entire namespace.

match.context

There are four possible values: ANY (global), SIDECAR_INBOUND (sidecar inbound), SIDECAR_OUTBOUND (sidecar outbound), GATEWAY (gateway).

listener.filterChain.filter

Specifies the filter in the filter chain; here it points to Envoy’s HTTP filter.

The relevant fields can be seen in the figure below:

In a Sidecar or Gateway, some metrics are exposed locally, and Prometheus can be configured to scrape them.

If you need to access the relevant metrics, you can use the following endpoint:

1
curl 127.0.0.1:15020/stats/prometheus

At this point, however, the metrics you need may not be there, because Istio enables only a small subset of Envoy’s metrics to avoid overburdening the scraping side. To enable more metrics, you can do so by modifying the related configuration.

4.1 Three Rule Matching Modes

Istio provides three rules for filtering metrics:

inclusionRegexps, enable metrics matched by regular expression
inclusionPrefixes, enable metrics matched by prefix
inclusionSuffixes, enable metrics matched by suffix

4.2 Two Configuration Levels

Global level

Modify the mesh field in the istio ConfigMap

1
kubectl edit cm istio -n istio-system

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
kind: ConfigMap
apiVersion: v1
metadata:
  name: istio
  namespace: istio-system
data:
  mesh: |-
    defaultConfig:
      proxyStatsMatcher:
        inclusionRegexps:
          - ".*http.*"
      gatewayTopology:
        numTrustedProxies: 1    

This enables all HTTP-related metrics; restart the Istio Gateway for it to take effect. In the test environment, the data scraped by Metrics jumped from 90MB to 140 MB right away.

Pod level

1
2
3
4
5
6
7
8
9
apiVersion: v1
kind: Pod
metadata:
  name: test
  annotations:
    proxy.istio.io/config: |-
      proxyStatsMatcher:
        inclusionRegexps:
          - ".*http.*"      

This applies only to pods with the sidecar injected.

5. Summary

This article mainly introduced the basic concepts and usage of EnvoyFilter, along with EnvoyFilter-related metrics monitoring.

6. References

https://istio.io/latest/docs/reference/config/networking/envoy-filter/