Cloudflare request shadowing 🥷🚧

Transparently send requests from one URL to another with output comparison.

Test for API compatibility, of any size change, to HTTP endpoints without expensive manual testing or disrupting production traffic. This is intended to complement automated testing suites by bringing that extra few feet of confidence in changes.

overview.mp4

Summary

• Little to no overhead 🤝
  • Shadowing occurs after the original request's response is sent back to the client--keeping your latency sensitive services happy
• Flexible configuration 🔨
  • Combine Workers' routes configuration and JavaScript
• At-rest data encryption 🔒
• Replay requests 🔁
• Automatic grouping 🥅
• Tagging 🏷️
• Export 📦
• Themed interface ☀️🌕
• Sharable links (group work in mind)

📋 / 📸

First class JSON diffs 👀

Compare JSON responses without inconsequential diffs.

• Objects: Properties can be moved but their value cannot change
• Arrays: Entries cannot move or change value
  • Moves are tracked separately from deletions/additions

Screenshots 📸

diff.mp4

Aggregation 📈

Visualize divergence trends with aggregated data through the UI or API.

Screenshots 📸

Automatic grouping 🥅

Quickly see what class of issue is happening most.

Groups are created for each unique set of divergent response keys. So, given:

• Response of shadow request A has 2 divergent keys name and price
• Response of shadow request B has 2 divergent keys name and price
• Response of shadow request C has 1 divergent key name
• Response of shadow request D has no divergent keys

We would have 2 groups:

• 🥐 Request A and B -- name and price
• 🥑 Request C -- name

Request D is not given a group or rendered on the page as it isn't divergent. It will be included in the aggregation graph under "Total"s though.

Screenshots 📸

Export 📦

Quickly export saved responses for use fixtures elsewhere.

Screenshots 📸

Tagging 🏷️

Apply tags you can filter by using the UI or API. Computed with JavaScript, you have the flexibility to create effective tags for your use-case.

Screenshots 📸

Sharable URLs

We try to make anything intractable translate to the URL so you can easily share what you're seeing with coworkers.

Privacy / encryption 🔑

Comfortably process requests knowing exactly what code is running with at-rest encryption* of sensitive content. Especially useful in regulated environments.

• Control
  • Request headers 🔐
  • Response body 🔐
  • Response headers are not saved
• Shadows
  • Request URL 🚫
    • Though encrypted in-transit by TLS, we consider URLs as low sensitivity content and save it in plain-text. Do not put sensitive content in URLs!
  • Request method 🚫
  • Request headers 🔐
  • Response body 🔐🚫
    • We save which paths diverge in plain-text for performant lists and grouping. Everything else is encrypted.
      • For example, if the control response and shadow response's .name properties diverge, ['name'] would be saved in plain-text while the full value is encrypted.
  • Response headers 🔐
  • Response status code 🚫
• Tags 🚫

See schema/table for a rough idea on data structure

* Using a 256 bit AES-GCM key derived, from a secret of your choice, using PBKDF2. See source code for implementation.

Replays 🔁

Systems can be complex and indeterminate. Replays allow you to resend requests ad-hoc to help track down flaky mismatches.

Replays trigger a request to the same URL and headers that triggered the original shadow. This triggers a shadow as usual but the result will be saved to the shadow you triggered the replay from instead of creating a new one.

Screenshots 📸

Light and dark themes

Supporting both people who like to actually read whats on their monitor with bright lights around and those who won't accept anything but a dark mode (or to avoid late night flash bang outs)

Page theme follows system/browser theme

Screenshots 📸

Deployment 🚢

Note

You'll need to use Cloudflare as a reverse proxy¹ to run this!

You are responsible for deploying and operating this tool. I'll do what I can to answer questions and provide guides though. 🙂

There are 3 runtime components:

• shadower: Forwards original (aka control) requests, sends shadow request, compares responses, and saves output to database
• api: Pull records from database
  • Note: I had originally only separated this as Pages' Workers struggled with Node.js compatibility on (for pg module) which may no longer be the case. At this point, I like the boundary.
• web: Front-end to api rendering diffs (skip for bring-your-own-interface)

How to

What to bring:

• Postgres server
  • Any reasonably recent version should do
    • jsonb + its operators are the only "hasn't been in Postgres for a few decades" features in use
  • Sizing is relative to expected load
    • Anecdotally: We've been running AWS' Aurora Serverless with 4 APU at 4/rps (20/rps burst) without breaking 25% database load.
• Cloudflare account

Steps:

1. Git clone or download project
2. Setup Cloudflare Access for the domain you'll host the web interface on
   • This will likely be project-name.pages.dev where project-name is in deploy of these scripts
3. Create database, table, and user
4. npm ci: Install setup script dependencies
5. node setup.mjs: Run setup script
   • Alternatively, do what setup.mjs is doing by hand
6. Adjust getShadowingConfigForUrl in shadower
   • See option documentation under ShadowingConfig type
7. Adjust routes in shadower wrangler.toml
8. Deploy shadower (npm run deploy in shadower)

Database

1. Create database: CREATE DATABASE request_shadowing
2. Verify gen_random_uuid is available: SELECT gen_random_uuid();
   • Enable uuid-ossp if not: CREATE EXTENSION uuid-ossp
3. Create tables and indices
   • Run tables.sql
   • Open up an issue with your use-case and index if you end up adding your own/replacing the out-of-box ones! 💙
4. Create user
   • Least amount of privileges possible. It doesn't do anything special.
5. Setup permissions for user
   • GRANT SELECT, INSERT, UPDATE, REFERENCES ON requests TO user;
6. Good to go!

Footnotes

1. Verify there is an "orange cloud" on the dashboard for the domain you intend to use. See docs/orange-cloud.png. ↩