Do capability restriction on a per-plugin basis

[nullpo-head]

The capability restriction system introduced in #89 is currently working on a per-VM basis, but it should be done on a per-plugin basis because users can provide capability restriction configurations for each plugin, not for each VM.
Let's make the capability restriction work on a per-plugin basis. The implementation design is to be discussed in the comments.

[nullpo-head]

@PiotrSikora , I've looked at the cpp-host code base, and my first thought was to have vm_key contain a capability config so that plugins that have different capability configs run in different VMs.
However, I heard from @mathetake that you have concern about that design. I really appreciate it if you could elaborate on your concern.

[PiotrSikora]

You could configure different capabilities for different plugins in the same VM, but you'd have hard time enforcing some of the restrictions.

I believe the enforcement would work fine on a per-plugin basis for callbacks and hostcalls bound to a given context (e.g. proxy_on_http_request_headers, proxy_get_header_map_pairs).

However, for independent hostcalls (e.g. proxy_http_call), you cannot prevent plugins from colluding to bypass the restrictions put on one of them, but not the other. For example, if you have 2 plugins configured in the same VM such as:

• plugin "A": allowed calls: proxy_on_http_request_headers, proxy_get_header_map_pairs,
• plugin "B": allowed calls: proxy_on_http_request_headers, proxy_get_header_map_pairs, proxy_http_call,

then plugin "A", which cannot make outgoing HTTP calls during its callback execution, could internally queue HTTP calls that plugin "B" would dispatch during its callback execution, bypassing capability restrictions.

I don't have an easy answer for how to fix that right now.

[nullpo-head]

My idea above is basically to make different plugins with different capabilities always run in different VMs, not in the same VM. If they run in different VMs, they cannot collude or violate the data of each other.

However, I agree with you that it's difficult to enforce different capabilities in single VM. In addition, I feel even the restrictions of callbacks and hostcalls theoretically can be bypassed. Suppose that you have 2 plugins with different capabilities such that

• The wasm code of which is basically an interpreter of a script language, such as mRuby or Python
• Users provide different source codes of that script language as its actual filter functions via plugin configs

In that case, since those plugins share the same linear memory in the same VM, they can maliciously rewrite each other's script source code. In other words, they can exploit the capabilities that other plugins have. This is an artificial example, but I think it shows that theoretically, as long as plugins with different capabilities run in the same VM, the capability restriction can be circumvented. So, I suggest to make them run in different VMs. Please correct me if my understanding is wrong, especially because I'm new to the code base yet ;P

[nullpo-head]

@PiotrSikora What do you think about the idea of running plugins with different capability configs in different VMs?

[PiotrSikora]

What you're suggesting (running plugins in diffrent VMs if they require different capabilities) is quite different than enforcing capability restrictions on a per-plugin basis for plugins running in the same VM. Also, it's effectively only enforcing capability restrictions on a per-VM basis (which we do now), but scheduling plugins in different VMs instead of using the same VM.

However, the ability to run multiple plugins in the same VM is a feature (it can be used to improve code sharing and decrease total bytecode size across plugins, save total memory usage, improve cross-plugin data sharing, etc.), so we cannot run them in different VMs without explicit opt-in.

Since we cannot run plugins in multiple VMs, and as I mentioned before, I don't think that we can realistically do the capability restrictions on a per-plugin basis for multiple plugins running in the same VM, the only thing that we can do is to enforce that the same set of capability restrictions is configured for all of them, and rejecting configuration otherwise. We should probably move capability_restriction_config to vm_config in Envoy (note: I imagine that this isn't foolproof right now, and having different vm_config values might result in multiple VMs being started, but that should be considered a bug).

We actually need to do this for a few more things that require stable sandbox. For example, WASI environment variables and preopened files/directories are loaded as part of the global constructors, so they are bound to the VM and not to a particular plugin.

[nullpo-head]

We should probably move capability_restriction_config to vm_config in Envoy.

I'm all for this idea, if having a capability_restriction_config for each plugin is not a requirement. It seems the most reasonable choice to me. If you decide to go that route, I'll close this issue, and open an issue for that in Envoy side.

JFYI, my rationale for my original idea is as follows.

What you're suggesting (running plugins in diffrent VMs if they require different capabilities) is quite different than enforcing capability restrictions on a per-plugin basis for plugins running in the same VM. Also, it's effectively only enforcing capability restrictions on a per-VM basis

Yes, you're right. That is what I suggested, and I thought it was a reasonable design choice.

However, the ability to run multiple plugins in the same VM is a feature ,(...) so we cannot run them in different VMs without explicit opt-in.

Being able to run plugins the same VM is a feature, but according to the doc, the feature is effective only when
All plugins which use the same vm_id and code will use the same VM

So, I thought it's reasonable to add a condition that and the same capability config here as well.
However, as I said already I agree with just moving the capability config to the vm config, because it's simpler.

[PiotrSikora]

I'm all for this idea, if having a capability_restriction_config for each plugin is not a requirement. It seems the most reasonable choice to me. If you decide to go that route, I'll close this issue, and open an issue for that in Envoy side.

Let's do that. Thanks! @mathetake any objections/comments?

Being able to run plugins the same VM is a feature, but according to the doc, the feature is effective only when
All plugins which use the same vm_id and code will use the same VM

The only purpose of vm_id is to isolate execution across different VMs.

[mathetake]

Let's do that. Thanks! @mathetake any objections/comments?

No objection! I think we've reached consensus now. Thanks!