Indexing: Introduce new experimental indexer API - NESTED PR!

[JReinhold]

Changelog

This PR contains multiple changes that should be referenced like so in the changelog (replacing the entry that this PR creates):

• Indexer: Introduce new experimental indexer API - Indexing: Introduce new experimental indexer API - NESTED PR! #23691, thanks @JReinhold!
• Addon-docs, Core, Server: Use new indexer API - Core, Server, Addon-docs: Replace story indexers with new API #23660, thanks @JReinhold!
• Server: Add support for tags - Core, Server, Addon-docs: Replace story indexers with new API #23660, thanks @JReinhold!
• Core-server: Improve internal types - Core: Improve internal types #23632, thanks @JReinhold!

---

Completes milestone 1 of #23457

These changes have already been reviewed separately in:
#23582
#23626
#23656
#23660
#23676
#23632

What I did

How to test

Checklist

• Make sure your changes are tested (stories and/or unit, integration, or end-to-end tests)
• Make sure to add/update documentation regarding your changes
• If you are deprecating/removing a feature, make sure to update
  MIGRATION.MD

Maintainers

• When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli/src/sandbox-templates.ts
• Make sure this PR contains one of the labels below.

["cleanup", "BREAKING CHANGE", "feature request", "bug", "build", "documentation", "maintenance", "dependencies", "other"]

🦋 Canary release

This pull request has been released as version 0.0.0-pr-23691-sha-8f3fbd86. Install it by pinning all your Storybook dependencies to that version.

More information

Published version	0.0.0-pr-23691-sha-8f3fbd86
Triggered by	@JReinhold
Repository	storybookjs/storybook
Branch	indexer-api
Commit	8f3fbd86
Datetime	Wed Aug 2 22:13:09 UTC 2023 (1691014389)
Workflow run	5744299829

To request a new release of this pull request, mention the @storybookjs/core team.

core team members can create a new canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=23691

[shilman]

Looks great!

[thebuilder]

Upgraded to Storybook 7.3.0, and all my component stories now fail to indexed, crashing the storybook build. Related to this PR?

WARN 🚨 Unable to index files:
WARN - ./src/components/Anchor/Anchor.story.tsx: Invariant failed: No matching indexer found for /Project/Frontend/src/components/Anchor/Anchor.story.tsx
...

My main.ts:

import { StorybookConfig } from '@storybook/react-vite';
import { projectConfig } from '../src/project-config';

// Set an env variable, so we can detect Storybook inside other tools (Mainly Tailwind)
process.env.STORYBOOK = 'true';

const config: StorybookConfig = {
  framework: '@storybook/react-vite',
  staticDirs: [
    '../public',
    {
      // Move the favicon.svg to the root, so it can be found by the browser
      from: '../public/assets/favicon.svg',
      to: '/favicon.svg',
    },
  ],
  stories: [
    {
      directory: '../docs',
      titlePrefix: 'Docs',
      files: '**/*.mdx',
    },
    '../src/**/*.@(story|stories).@(js|jsx|ts|tsx)',
  ],
  addons: [
    '@storybook/addon-links',
    {
      name: '@storybook/addon-essentials',
      options: {
        backgrounds: false,
      },
    },
    '@storybook/addon-a11y',
    '@storybook/addon-interactions',
    '@storybook/addon-designs',
  ],
  core: {
    builder: '@storybook/builder-vite',
  },

};

export default config;

[shilman]

@thebuilder confirmed this is a bug. should have a fix shortly. sorry for the inconvenience!

[shilman]

@thebuilder fixed in https://github.com/storybookjs/storybook/releases/tag/v7.3.1

[GreenGremlin]

I'm still seeing this error on v7.3.1

[kasperpeulen]

I'm still seeing this error on v7.3.1

Could you share your main.js file? @GreenGremlin

[GreenGremlin]

@kasperpeulen here's our main.ts file:

import type { StorybookConfig } from '@storybook/react-webpack5';
import childProcess from 'child_process';
import path from 'path';
import webpack, { type Configuration } from 'webpack';

import buildAliasesFromTsconfig from '../config/buildAliasesFromTsConfig';

// This allows us to run storybook from a single package or for the entire repo
function getStoriesGlobs() {
  if (!process.env.PWD) {
    throw new Error(
      'process.env.PWD not set! This is typically set by node, but could have been modified after the node process was started.'
    );
  }
  const baseUrl = path.relative(__dirname, process.env.PWD);
  if (baseUrl === '..') {
    return [baseUrl + '/packages/**/src/**/stories.@(ts|tsx)', baseUrl + '/packages/**/src/**/*.stories.@(ts|tsx)'];
  }
  return [baseUrl + '/src/**/stories.@(ts|tsx)', baseUrl + '/src/**/*.stories.@(ts|tsx)'];
}

const commitHash = childProcess.execSync('git rev-parse --short HEAD').toString().trim();
const isIntegrationBuild = process.env.IS_INTEGRATION_BUILD === 'true';
const isProductionBuild = process.env.NODE_ENV === 'production';
const isSnapshotBuild = process.env.IS_SNAPSHOT_BUILD === 'true';

const storybookConfig: StorybookConfig = {
  framework: {
    name: '@storybook/react-webpack5',
    options: {
      builder: { lazyCompilation: true },
    },
  },
  core: {
    // Our storybook setup is likely too custom for these reports to be useful.
    disableTelemetry: true,
    enableCrashReports: false,
  },
  features: {
    // Bundle splitting causes percy to fail with "TypeError: Cannot read property 'message' of undefined".
    // The actual error is difficult to access, but it appears to be an uncaught error in a promise in the browser.
    // This error only repros on CircleCI.
    storyStoreV7: !isSnapshotBuild,
  },
  stories: getStoriesGlobs(),
  addons: [
    '@storybook/addon-a11y',
    '@storybook/addon-essentials',
    '@storybook/addon-knobs',
    '@storybook/addon-links',
    'storybook-addon-designs',
  ],
  docs: {
    autodocs: true,
  },
  webpackFinal: (config: Configuration) => {
    // Slight hack to fix "Unable to determine deployment - Unexpected hostname" error when loading stories that use
    // components that load the configuration manager.
    config.plugins = config.plugins?.concat(
      new webpack.DefinePlugin({
        __DEV__: true,
        __INTEGRATION_BUILD__: isIntegrationBuild,
        __SNAPSHOT_BUILD__: isSnapshotBuild,
      })
    );
    config.resolve ??= {};
    config.resolve.alias = {
      ...config.resolve.alias,
      '@internal-test/giraffe-client': path.resolve(
        __dirname,
        '..',
        'packages/basics/giraffe-client/src/utils/testhelpers/index'
      ),
      '@internal-test/internal-user-utils': path.resolve(
        __dirname,
        '..',
        'packages/basics/internal-user-utils/src/utils/testhelpers/index'
      ),
      // jsdom is not needed in storybook so far and this empty mock prevents it for using NodeJS modules that break the storybook build
      jsdom: path.resolve(__dirname, 'jsdomMock.ts'),
      ...buildAliasesFromTsconfig(),
    };

    config.resolve.extensions ??= [];
    config.resolve.extensions.push('.ts', '.tsx');

    config.resolve.fallback = {
      fs: false,
    };

    // Append commitHash to JS files instead of contentHash to help serve on engdocs
    // Engdocs has multiple pods, each of which creates its own set of assets.
    // If the bundle names differ, it will fail to serve depending on how the request is routed
    config.output ??= {};
    config.output.filename = `[name].${commitHash}.js`;
    config.optimization ??= {};
    config.optimization.chunkIds = 'deterministic';

    // Addresses OOM issues
    config.optimization.minimize = false;
    config.devtool = false;

    config.module ??= {};
    config.module.rules ??= [];

    const tsRuleIndex = config.module.rules.findIndex(
      rule => typeof rule === 'object' && 'test' in rule && rule.test instanceof RegExp && rule.test.test('test.ts')
    );
    if (tsRuleIndex === -1) {
      throw new Error("No *.ts rule found in storybook's webpack config.");
    }
    // Replace storybook's babel-loader with esbuild-loader. Matches the config from webpack.config.base.ts
    config.module.rules.splice(
      tsRuleIndex,
      1,
      {
        // Storybook includes some .mjs files.
        test: /\.m?js$/,
        exclude: [/node_modules/],
        enforce: 'pre',
        use: ['source-map-loader'],
      },
      {
        test: /\.tsx?$/,
        exclude: [/node_modules/],
        use: [
          {
            loader: 'esbuild-loader',
            options: {
              target: 'es2018',
              // Setting this to 'automatic' currently breaks storybook builds.
              // For some reason jsx_dev_runtime.jsxDEV is undefined when jsx = 'automatic'.
              jsx: undefined,
              // injects filename and source location into each JSX element, great for debugging
              jsxDev: !isProductionBuild,
            },
          },
        ],
      }
    );

    return config;
  },
};

export default storybookConfig;