Selecting a list of WASM filters that a request will run through based on domain/host

[marc-barry]

Title: Question about how to select a list of WASM filters that a request will run through based on domain/host.

Description:

I have a need to proxy plaintext HTTP and dynamically select the list of WASM filters a request will go through based on the domain/host. The HTTP connection manager (https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto.html#http-connection-manager-proto) only supports a single list of filters. I do have virtual hosts (https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto) which match based on domain and cannot use the typed_per_filter_config as this isn't supported for WASM filters as per #26440.

To better explain what I'd like to do is the following:

• www.example.com
  • WASM filter 1
  • WASM filter 2
  • WASM filter 3
• example.com
  • WASM filter 1
• github.com
  • WASM filter 2
• api.github.com
  • WASM filter 4

I attempted to use a composite filter https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/composite_filter but it is designed to match based on the matching API and select a single filter. The list of domains and configurations above is simplified as I'm dealing with 100's to 1000's of domains and have many WASM filters. I am aware that I can match based on the :authority header but I can't direct that to a filter list.

I'm reaching out to the community to see if there is a way to accomplish what I'm trying to do with WASM filters in the absence of not being able to support it on the per route basis like other filters can.

Relevant Links:

• enable Wasm filter enable per route #26440
• Support Wasm Specific Route  #16651
• Support for Per Route in Wasm Plugin proxy-wasm/proxy-wasm-cpp-host#188
• Support Wasm Specific Route  #16651 (comment)
• enable a disabled filter by complete empty config or meaningless placeholder in the route #31482

[marc-barry]

I also found #16651 which seems to indicate that it isn't supported yet. I also found proxy-wasm/proxy-wasm-cpp-host#188.

[marc-barry]

A large list of issues which might be related: #16651 (comment)

[marc-barry]

The error message you get when you try to use it per filter is:

"The filter envoy.filters.http.wasm doesn't support virtual host or route specific configurations".

[wbpcode]

cc @mpwarres

[wbpcode]

Does this make sense? https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/http_filters#route-based-filter-chain

By that way, you can disable all wasm filters by default in the HCM and them enable specific some wasm filters on specific virtual host.

Considering that the wasm filter has no per route level config, you can use a hack way to enable the filter.

  # See https://github.com/envoyproxy/envoy/issues/31482 
  typed_per_filter_config:
   wasm_x:
      "@type": type.googleapis.com/envoy.config.route.v3.FilterConfig
      config: # Note this config field could not be empty because the xDS API requirement.
        "@type": type.googleapis.com/google.protobuf.Empty  # Empty as a placeholder.
      is_optional: true

[marc-barry]

Does this make sense? https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/http_filters#route-based-filter-chain

@wbpcode this does make sense. I managed to get all my filters loaded in with disabled: true and they are of this form:

{
    "http_filters": [
        {
            "name": "envoy.filters.http.wasm",
            "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                "config": {
                    "name": "wasm1",
                    "vm_config": {
                        "vm_id": "default",
                        "runtime": "envoy.wasm.runtime.v8",
                        "code": {
                            "local": {
                                "filename": "/app/wasm/wasm1.wasm"
                            }
                        }
                    },
                    "configuration": {
                        "@type": "type.googleapis.com/google.protobuf.StringValue",
                        "value": "{}"
                    }
                }
            },
            "disabled": true
        },
        {
            "name": "envoy.filters.http.wasm",
            "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                "config": {
                    "name": "wasm2",
                    "vm_config": {
                        "vm_id": "default",
                        "runtime": "envoy.wasm.runtime.v8",
                        "code": {
                            "local": {
                                "filename": "/app/wasm/wasm2.wasm"
                            }
                        }
                    },
                    "configuration": {
                        "@type": "type.googleapis.com/google.protobuf.StringValue",
                        "value": "{}"
                    }
                }
            },
            "disabled": true
        }
    ]
}

But I can't figure out how to enable them. I've added a typed_per_filter_config to the routes of a virtual host with a type of type.googleapis.com/envoy.config.route.v3.FilterConfig keyed by the names of the filters (i.e. wasm1, wasm2...) but it doesn't enable the filters for those routes.

[marc-barry]

For completeness my typed_per_filter_config looks like this:

{
    "typed_per_filter_config": {
        "wasm1": {
            "@type": "type.googleapis.com/envoy.config.route.v3.FilterConfig",
            "config": {
                "@type": "type.googleapis.com/google.protobuf.Empty"
            },
            "is_optional": true
        },
        "wasm2": {
            "@type": "type.googleapis.com/envoy.config.route.v3.FilterConfig",
            "config": {
                "@type": "type.googleapis.com/google.protobuf.Empty"
            },
            "is_optional": true
        }
    }
}

[marc-barry]

I see warnings like this in the logs:

1.7143545561019342e+09	info	[2024-04-29 01:35:56.101][19][warning][http] [source/common/router/config_impl.cc:2199] Can't find a registered implementation for http filter 'wasm1' with type URL: 'google.protobuf.Empty'
1.7143545561019516e+09	info	[2024-04-29 01:35:56.101][19][warning][http] [source/common/router/config_impl.cc:2199] Can't find a registered implementation for http filter 'wasm2' with type URL: 'google.protobuf.Empty'

[wbpcode]

cc @marc-barry the used keys in the typed_per_filter_config should be names in the http_filters. So, I think you need to do two things:

1. use different names in the http_filters for wasm filter. (we will load the filter according the type rather than the name).
2. use the names as the keys of typed_per_filter_config.

[marc-barry]

@wbpcode thanks for clarifying. It works. I apologize for the misunderstanding of how keys work in the typed_per_filter_config. I was confused in my thinking that the config name that was the key. But in hindsight, it is logical that it behaves in the manner you pointed out. Thanks for all your help. I have about 30 WASM filters now in a disabled state by default and then dynamically enable them via the typed_per_filter_config. I hope that isn't too much of a performance impact as I imagine Envoy needs to iterate over all the filters with an O(N) operation. One day I expect to have potentially have > 100 filters. Hopefully this isn't an issue and that one day #26440 is an option for my use case.

[marc-barry]

Since @wbpcode gave me a solution to my issue I'm going to close this. I'll keep an eye on the other tickets for when #26440 is completed and options within #31482 are possibly considered to make the options more clear. Thanks for the help.