How to measure CPU time?

[alexandernst]

Since CF has 10ms cpu-time limit (50ms for paying customers), I'd like to be able to measure the cpu-time the exact same way CF will measure it once my code is deployed. Is it possible to do such a measurement with miniflare?

[mrbbot]

Hey! 👋 You may be able to do this using the profiler built into the Chrome DevTools. See https://miniflare.dev/developing/debugger#nodejs-inspector for setup instructions, then click the Profiler tab at the top of popup window. This measurement will probably be very different than the deployed though, as the implementations of the underlying APIs and the hardware the code is running on will be completely different. Could still be useful for measuring relative performance changes between different versions of your worker.

[alexandernst]

This is not helpful at all. Can we keep the issue open until there is a way to measure the execution time in a manner close to what CF does. I believe this is an important topic since this is a fundamental part of how CF bills and how code is decided wether it should be killed or not.

[mrbbot]

Apologies, I should've added some more detail to my previous answer. The problem here is it's very difficult to get a useful measurement in a manner close to Cloudflare.

Cloudflare creates a new V8 isolate for each worker. Isolates have nice APIs for measuring CPU time. However, Miniflare uses Node.js which, whilst it also runs on V8, doesn't yet expose the isolate API. It does let you create a new V8 context (what Miniflare uses for sandboxing), but this runs in the same isolate as the main entry context.

We could approximate the CPU time by timing each request as we do right now, and subtracting async I/O time (by recording how long fetchs take for instance), but this will always give different results to Cloudflare as we're not running on the same server hardware, and the Miniflare implementations of runtime APIs are less performant, running in JavaScript as opposed to optimised C/C++. It would be interesting to see how big this difference is, it's possible it's neglible.

I agree with you this would be a nice feature to have, I just don't think it's possible to do it in a way that's useful.

[alexandernst]

I wasn't expecting a measurement near equal to what CF does, but here's why I'm asking for this:

I started working on a project. "Nothing too CPU-expensive" I thought. Just some JSON parsing, some operations with Web Crypto, and a little bit of business logic here and there (this field in the JSON shouldn't be equal to XYZ, this other field should be an array, if this other field is "0" then do ABC...).

I tested the code with miniflare and everything was working perfectly. "Time to deploy" I thought... Just to find out that I was exceeding the 10ms limit of the free tier[0]. So I upgraded to paid workers (50ms) and now my code runs successfully ~25% of the times, which means that my code fits the 50ms budgets sometimes.

I found out that my project wasn't ever going to work on CF Workers (because of CPU constraints) just after reaching the "deploy" stage, which is pretty terrible. What is even worse, I don't even know how much CPU time is it using, so I can't say if investing some more time in optimizing somehow my code will make it fit in the 50ms budget, or if I should totally abandon the idea and just go with some other serverless service.

What I'm trying to say is that while I perfectly understand that miniflare just can't provide CF-level measurements, it would be extremely helpful if it could provide approximate measurements. Anything in the range of <30% error margins would still be very useful.

[0] - I'm sure it's not memory-limits related, but it would be super useful if miniflare could also tell me how much memory did my request use.

[ffflabs]

Hey @alexandernst it sounds like it would be more straightforward to request help on the Discord Server https://discord.com/channels/595317990191398933/846453104382836766. There's a chance you could pinpoint your performance issue directly (through a CF representative) instead of benchmarking a sandbox.

[alexandernst]

@ffflabs IMHO that is a really bad idea. I (or any other developer using CF's workers) shouldn't depend on the willingness of CF employees to debug my code and/or give me tips about what could be wrong with it. Instead, I should have proper tools to debug the problem on my own.

[y21]

Not sure if this helps at all, but Node.js exposes APIs for measuring CPU time (sort of). process.cpuUsage() returns CPU time used by the process, in microseconds. It's possible to implement this in userland code (i.e. worker code running in miniflare), but that's really hacky because you need to escape the VM sandbox to get a reference to the host process object. This works for me:

const process = this.constructor.constructor("return process")();

addEventListener("fetch", (event) => {
  const timer = process.cpuUsage();
  
  let t = Date.now();
  // waste 100ms
  while (Date.now() - t < 100) ;

  event.respondWith(new Response("Hello Miniflare!"));
  console.log("cpu time", process.cpuUsage(timer));
});

As mentioned above by others, depending on your hardware, this value could be way off from what you'd get when your script runs on CF workers. Node also uses uv_getrusage() from libuv for process.cpuUsage() and I don't know how exactly CF workers measures CPU time, but this might also be a source of inconsistency.

[mrbbot]

@y21 Ooo that's cool, didn't know about process.cpuUsage(), that could definitely work for an approximate measurement. Nice sandbox escape too... 😅