Support Rich HTTP Probe Types via Headless Chrome

[conallob]

Low-level HTTP requests are straightforward to create using external probes. However, rich web apps leveraging asynchronous Javascript, etc are harder to test using low-level HTTP probes. To work around these limitations, probe developers often end up straying from E2E testing that mimics user queries and use synthetic endpoints that can be probed using low-level HTTP requests.

To better support E2E probes of rich HTTP endpoints, would it be possible to extend Cloud Prober to also support Headless Chrome/Chromium probe types? I'm envisioning a configuration that maps to Headless Chrome/Chromium configuration options.

Off the top of my head, it may be helpful to leverage existing open source products like Vanadium, Chromium's WebView and/or Puppeteer / Puppetry

[manugarg]

I've been thinking of adding this as well. I was looking at selenium and playwright for this kind of testing. Both of them provide rich browser testing - open the website, fill in some fields, interact with the elements (click, hover on, etc), verify certain elements exist, etc. Both of them don't provide any Go bindings, so executing an external program is the easiest option in that regard. AWS canary checks also use selenium wrappers to run such tests -- I believe it allocates/finds a VM and runs the test there.

Just FYI, I found this Cloudprober external prober:
https://github.com/nmcclain/cloudprober_external_playwright
It runs the playwright command to execute the actual tests and it expects tests to be predefined.

I think at a minimum if we could provide a way to define a test through configuration, we can then run a test anywhere (say invoking a test on some shared service). I'll do more research.

[manugarg]

Update: I've given it more thought and have been working on a possible solution using cypress. Any UI testing tends to be pretty heavy, so I am thinking of using a model where UI testing functionality runs as an independent server alongside Cloudprober (in k8s deployments you can think of it as a sidecar) and Cloudprober sends a request to this service to run UI probes.

[Cloudprober] <===(on probe)===> [UI test server]

I'll share more details and working code soon.