Tinybench runs the test cases in parallel which is bad for tests that access resources like files

[BioPhoton]

Problem:
I wrote a benchmark to compare file crawling and get quite different results compared to e.g. benchmark. With benchmark I'm able to run tests in sequence so they can all after another access the same resources. In this way the results are reliable.

With tinybench I throw out of memory or other odd errors due to parallel processing.

Suggested solution:

Implement a way to run all test cases in sequence.

Links:

• Promise support bestiejs/benchmark.js#176 (comment)

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[Aslemammad]

Sure, @sirenkovladd might be interested?

[peterHakio]

Also relevant in context of vitest vitest-dev/vitest#5412

[Aslemammad]

Tasks are already running sequentially. https://github.com/tinylibs/tinybench/blob/main/test/sequential.test.ts

So let me know what is the specific issue! As far as I know, vitest does not use bench.run from tinybench and it implements its own way.

[Aslemammad]

@BioPhoton Do you have any reproduciton?

[BioPhoton]

I have the following suite:

// eslint-ignore-next-line import/no-named-as-default-member
import fastGlob from 'fast-glob';
import { glob } from 'glob';
import { globby } from 'globby';
import { join } from 'node:path';
import yargs from 'yargs';

const cli = yargs(process.argv).options({
  pattern: {
    type: 'array',
    string: true,
    default: [
      join(process.cwd(), 'node_modules/**/*.md'),
    ],
  },
  outputDir: {
    type: 'string',
  },
  logs: {
    type: 'boolean',
    default: true,
  },
});

// eslint-disable-next-line n/no-sync
const { pattern, logs } = cli.parseSync();

if (logs) {
  // eslint-disable-next-line no-console
  console.log('You can adjust the test with the following arguments:');
  // eslint-disable-next-line no-console
  console.log(
    `pattern      glob pattern of test      --pattern=${pattern.toString()}`,
  );
}

const fastGlobName = 'fast-glob';
const globName = 'glob';
const globbyName = 'globby';

// ==================

const suiteConfig = {
  suiteName: 'glob-matching',
  cases: [
    [fastGlobName, callAndValidate(fastGlob.async, pattern, fastGlobName)],
    [globName, callAndValidate(glob, pattern, globName)],
    [globbyName, callAndValidate(globby, pattern, globbyName)],
  ]
};
export default suiteConfig;

// ==============================================================
const logged: Record<string, boolean> = {};
function callAndValidate<T = string | string[]>(
  fn: (patterns: T) => Promise<unknown[]>,
  globPatterns: T,
  fnName: string,
) {
  return async () => {
    const result = await fn(globPatterns);
    if (result.length === 0) {
       throw new Error(`Result length is ${result.length}. This test is not meaningful`);
    } else {
      if (!logged[fnName]) {
        logged[fnName] = true;
        console.log(
          `${fnName} found ${result.length} files for pattern ${pattern.join(
            ', ',
          )}`,
        );
      }
    }
  };
}

Somehow it produces different test results than expected.

[Aslemammad]

Ok, I'll check this out soon.