AssetNotFoundException running custom builder

[samdeane]

We have a custom builder which emits a file for each element that has a certain annotation, and then another builder which consumes those files to build an index function.

We are sometimes hitting AssetNotFoundException with the second builder.

It seems to be some sort of race condition relating to the generated index file. I suspect this because it doesn't always happen, and a second run after a failure will always succeed (presumably because the index file is missing by then).

Info

Dart version: Dart SDK version: 2.19.3 (stable) (Tue Feb 28 15:52:19 2023 +0000) on "macos_arm64"

Build: 2.3.1

full pubspec.lock

In order to route, prioritize, and act on this as soon as possible please include:

OS: macOS, building to iOS

Error Log

You have hit a bug in build_runner
@club/client-mobile:generate: Please file an issue with reproduction steps at https://github.com/dart-lang/build/issues
@club/client-mobile:generate: 
@club/client-mobile:generate: 
@club/client-mobile:generate: AssetNotFoundException: club_view|lib/view_index.dart
@club/client-mobile:generate: package:build_runner_core/src/asset/file_based.dart 135:18           _fileForOrThrow.<fn>
@club/client-mobile:generate: dart:async                                                           _rootRunUnary
@club/client-mobile:generate: package:build/src/asset/reader.dart 53:13                            AssetReader.digest
@club/client-mobile:generate: package:build_runner_core/src/asset_graph/graph.dart 171:30          AssetGraph._setLastKnownDigests.<fn>
@club/client-mobile:generate: dart:async                                                           Future.wait.<fn>
@club/client-mobile:generate: package:build_runner_core/src/asset_graph/graph.dart 170:5           AssetGraph._setLastKnownDigests
@club/client-mobile:generate: package:build_runner_core/src/asset_graph/graph.dart 285:5           AssetGraph.updateAndInvalidate
@club/client-mobile:generate: package:build_runner_core/src/generate/build_definition.dart 466:5   _Loader._updateAssetGraph
@club/client-mobile:generate: package:build_runner_core/src/logging/logging.dart 25:18             logTimedAsync
@club/client-mobile:generate: package:build_runner_core/src/generate/build_definition.dart 221:21  _Loader.prepareWorkspace
@club/client-mobile:generate: package:build_runner_core/src/generate/build_impl.dart 114:27        BuildImpl.create
@club/client-mobile:generate: package:build_runner_core/src/generate/build_runner.dart 34:26       BuildRunner.create
@club/client-mobile:generate: package:build_runner/src/generate/build.dart 109:17                  build
@club/client-mobile:generate: package:build_runner/src/entrypoint/build.dart 35:18                 BuildCommand._run
@club/client-mobile:generate: package:args/command_runner.dart 212:13                              CommandRunner.runCommand
@club/client-mobile:generate: package:build_runner/src/entrypoint/run.dart 26:18                   run
@club/client-mobile:generate: .dart_tool/build/entrypoint/build.dart 60:16                         main
@club/client-mobile:generate: 

Viewer Source

The project itself is closed source, but here's the code for the viewer.

/// Extension of the file exported by the [ViewLocatingBuilder].
/// NOTE: this needs to match the `required_inputs` value
/// for the indexing builder, in build.yaml.
const velocityViewsExtension = '.velocityViews';

/// Index file built into `lib/` by the [ViewIndexBuilder].
const viewIndexFile = 'view_index.dart';

/// Builder which outputs a file for every source file
/// that contains at least one @VelcoityView annotation.
///
/// These files will be consumed by the [ViewIndexBuilder].
class ViewLocatingBuilder implements builder.Builder {
  @override
  final buildExtensions = const {
    '.dart': [velocityViewsExtension]
  };

  @override
  Future<void> build(builder.BuildStep buildStep) async {
    // print('starting view annotation generation');
    final resolver = buildStep.resolver;
    if (!await resolver.isLibrary(buildStep.inputId)) return;
    final lib = LibraryReader(await buildStep.inputLibrary);
    final viewAnnotation = TypeChecker.fromRuntime(VelocityView);
    final annotated = [
      for (var element in lib.annotatedWith(viewAnnotation))
        element.element.name,
    ];

    if (annotated.isNotEmpty) {
      await buildStep.writeAsString(
          buildStep.inputId.changeExtension(velocityViewsExtension),
          annotated.join(','));
    }
    // print('done view annotation generation');
  }
}

/// Builder which consumes the files output by the
/// [ViewLocatingBuilder].
///
/// This builder will output a single file, `lib/view_index.dart`,
/// which contains a `viewFactory` variable that can be used
/// to register all views in the application.
class ViewIndexBuilder implements builder.Builder {
  @override
  final buildExtensions = const {
    r'$lib$': [viewIndexFile]
  };

  @override
  Future<void> build(builder.BuildStep buildStep) async {
    print('starting view index generation');
    List<String> imports = [];
    List<String> files = [];

    final viewsFiles =
        buildStep.findAssets(Glob('**/*$velocityViewsExtension'));
    await for (var viewsFile in viewsFiles) {
      // get a relative import path, removing the leading 'lib/'
      final path = viewsFile.changeExtension('.dart').path.substring(4);

      // add the import to the list of imports
      imports.add('import \'$path\';');

      // read the file and add each view to the list of files
      final views = await buildStep.readAsString(viewsFile);
      for (var viewClass in views.split(',')) {
        files.add('  ${viewClass}(),');
      }
    }

    if (!files.isEmpty) {
      imports.sort((a, b) => a.compareTo(b));
      files.sort((a, b) => a.compareTo(b));

      final source = [
        '// GENERATED CODE - DO NOT MODIFY BY HAND',
        '',
        'import \'package:velocity/velocity.dart\';',
        '',
        ...imports,
        '',
        '/// Factory containing all views.',
        'final viewFactory = ViewFactory([',
        ...files,
        ']);'
      ];

      await buildStep.writeAsString(
        builder.AssetId(buildStep.inputId.package, 'lib/$viewIndexFile'),
        source.join('\n'),
      );
      print('done view index generation');
    }
  }
}

build.yaml

builders:
  componentIndex:
    import: 'package:velocity_generator/builder.dart'
    builder_factories: ['componentIndexBuilder']
    build_extensions: { '$lib$': ['component_index.dart'] }
    auto_apply: none
    build_to: source

  componentAnnotation:
    target: ':velocity_generator'
    import: 'package:velocity_generator/builder.dart'
    builder_factories: ['componentBuilder']
    build_extensions: { '.dart': ['.velocityComponent.g.part'] }
    auto_apply: none
    build_to: cache
    applies_builders: ['source_gen|combining_builder']

  viewAnnotation:
    import: 'package:velocity_generator/builder.dart'
    builder_factories: ['viewLocator']
    build_extensions: { '.dart': ['.velocityViews'] }
    auto_apply: none
    build_to: cache

  viewIndex:
    import: 'package:velocity_generator/builder.dart'
    builder_factories: ['viewIndexer']
    build_extensions: { '$lib$': ['view_index.dart'] }
    auto_apply: none
    build_to: source
    required_inputs: ['.velocityViews']

[jakemac53]

This exception actually looks like it is coming from a brand new run of build_runner - it is computing the new digests for all of the changed sources, but when it goes to do that the file does not exist.

This indicates that in between the initial computation of all the changed/added/removed sources, this file has been deleted. Do you have something that is manually deleting these view_index.dart files, that was possibly running at the same time as a build was starting?

[github-actions]

Without additional information we're not able to resolve this issue, so it will be closed at this time. You're still free to add more info and respond to any questions above, though. We'll reopen the case if you do. Thanks for your contribution!

[samdeane]

Hi, sorry, I missed the notification.

I don't believe that we have any manual process deleting that file.

[jakemac53]

Are you possibly running multiple instances of build_runner at a time? That could also cause this.

[samdeane]

Are you possibly running multiple instances of build_runner at a time? That could also cause this.

We're using turbo to drive some build things across multiple packages. I don't think we're parallelising the actual generation, but I will check, as it is possible.

Would it be a problem if build_runner was running simultaneously across different packages? It definitely shouldn't be running multiple times across the same package.

[jakemac53]

Running multiple times across different packages should be fine, unless maybe there are symlinks between them?

[jakemac53]

Without an open source reproduction case I don't think I can make any progress on this issue. Would it be possible for you to provide something? If not I think we will just have to close this as I unfortunately can't really take any action on it, and it doesn't do any good leaving it open forever.

[github-actions]

Without additional information we're not able to resolve this issue. Feel free to add more info or respond to any questions above and we can reopen the case. Thanks for your contribution!