Only scan the file that got changed and thus prevent build_runner from scanning unnecessary files #3448
Comments
|
build_runner already does dependency tracking in order to figure out which files it should run on based on a given change. The issue is that for any codegen which does analysis, we have to add a dependency on all transitively imported files, because even changes in transitive libraries could affect the current library (through inference and exposed types). Code generators can also reach the full transitive set. We also can't differentiate between API changes and method body changes today, because we do analysis from source and not summary files. This gives you access to the full AST (including method bodies). The best things you can do are use |
I've recently encountered with the same behavior as the OP, where a small change in an "unrelated" file would rerun the code generation for many files. I was wondering if this would also apply for the macros in metaprogramming or if it would use a different mechanism to invalidate less often. If it would be the same, I'm worried about how easy it would be to reach a point where a large portion of a big project is invalidated and builds get noticeable slower. |
The feature is carefully designed to enable much finer invalidation yes - but in practice it will depend on how much this is optimized by the implementations. Macros can only access the "api" of their dependencies, so for instance white space edits and edits to method bodies would not typically necessitate a re-run of macros in down stream dependencies. Type inference does complicate things a bit here though. The actual introspection API is also quite fine grained which would allow for further dependency tracking, if implementations decide it is worth the effort (and memory) to do such fine grained tracking. |
|
@jakemac53 Thank you for the insight |
|
Thanks for the help @jakemac53 - can you point to any material/documentation in terms of how build runner can be optimised? Currently we're a bit stuck, every time we save a single model/widget file (even one that does not need to be regenerated) it takes ~47 seconds on a M1 Pro, before the builder is done. I've tried as much as possible to limit using json_serializable:
generate_for:
- lib/common/domain/*.dartHowever, it still scans way too many files. And by the way, is it on the roadmap to somehow optimise this behaviour? E.g. coming from React world, I'm used to super quick hot-reloads, even in very large projects, when saving a single file, and I would love to see something similar for Dart, as development experience is seriously hindered for us currently. |
|
https://github.com/dart-lang/build/blob/master/docs/measuring_performance.md is the best place to start to understand where the time is going. Long term the goal is to integrate codegen into the compiler via macros which is how we will improve this significantly. If things are invalidating more than you expect it is most likely due to the import structure of your app. Any Dart file could be affected in arbitrary ways based on its full transitive imports, so the more specific your imports are the better (this applies to both build_runner and general hot reload). |
|
Thanks for that Jake! So when you say:
Do you mean making sure code is separated into smaller files? |
|
It is more about having fewer transitive imports - so called "convenience imports" in particular can lead to major pain. Those are imports which just export a whole bunch of other libraries, so you can import one thing and get everything. For small packages this is OK but for entire apps it means every library depends on the entire app, which will cause major scaling problems. |
|
If you are seeing files be regenerated that seem unrelated to the file you changed, it is almost certainly because of these types of import structures. A file must be regenerated if any transitive import is changed, whether it is directly referenced in the file or not. Changes to that file can still affect the analysis result indirectly. |
|
I see.. Well we usually have one file per class/widget. But I guess in terms of optimising the build runner, if we had two classes in the same file:
Where Class B depends on Class A, it wouldn't make any difference if we moved Class B into a separate file, since class B would have to import class A. Is that correctly understood? |
|
Correct, if there is a dependency between them then it is actually probably more efficient to keep them in a single library, to an extent. But I wouldn't recommend going very far down that path. The exact behavior here may differ between build_runner and regular hot reload (ie: frontend_server). In particular build_runner is purely file contents based and will invalidate based on even whitespace changes, while frontend_server is at least a bit smarter then that, and there is ongoing effort there to make it smarter. |
|
The larger value thing is not doing the |
|
Thanks a lot for your help Jake. Highly appreciated. Really looking forward to when build_runner is optimised - think it will have a huge impact on productivity |
|
A shoutout for this nice package that mitigates a bit the issue for now: https://github.com/jyotirmoy-paul/cached_build_runner @simplenotezy could you give it a try with your code sample? I have something similar on my end and it works quite well. Note: I had to do a build_runner build first to get cached_build_runner working. |
|
From a quick glance it looks like cached_build_runner is not taking transitive dependencies into account - that might be ok for a lot of cases but you may run into situations where it doesn't work. Depends a lot on the builder being used and the particular type of change in the transitive dep. So use at your own risk. |
|
Fwiw, you could also just make a different Resolver implementation that does the same thing (doesn't track transitive deps). While not safe/hermetic, it would probably give you much faster builds, and most of the time it probably does work out OK. I don't think it's something we can/should support directly but I have no problem with a third party implementation that is "unsafe" in this way. It wouldn't be super trivial to slot that resolver implementation in though (you would have to manually make a build script with build_runner_core). Possibly we could support a build.yaml option to automatically slot in a specified resolver implementation though (probably given a Uri to import, and constructor to invoke). |
|
Thanks I'll check out that package. For now our biggest development hurdle with Dart is the massive slowdown we experience mainly due to build_runner working 90% of the time (rebuilds take way too long!), so I really hope this is something Dart will look into soon |
|
Thanks for the explanations, I better understand the limits now. As you said, it could work on some specific situations and not on others. Will keep in mind limitations with transitives deps ; Still glad to see that the Flutter community is really reactive on those topics 👍 |
|
Hey @mikklfr thanks for the shoutout. I have implemented it in our code now, and will report back (here or in the repo) with my findings |
Question
Is there any way to only have it run the build_runner if an actual file was changed that requires the build_runner?
Problem: Changing this in my example widget, will take 10 seconds for the build_runner to process:
import 'package:my_app/common/exceptions/api_exceptions.dart'; import 'package:my_app/common/widgets/common/animated_loader.dart'; import 'package:my_app/common/widgets/common/error_handler.dart'; import 'package:my_app/common/widgets/test_widget.dart'; import 'package:flutter/material.dart'; import 'package:flutter_riverpod/flutter_riverpod.dart'; class AsyncValueWidget<T> extends StatelessWidget { const AsyncValueWidget({ super.key, }); @override Widget build(BuildContext context) { - return SizedBox(); + return Container(); }Details
I know it's been mentioned here: #2928 and many other places. I did configure the
generate_forinbuild.yamlHowever, I have an issue where if I save a widget file, e.g.
lib/common/widgets/async_value_widget.dartThen I'll see the following output in the build_runner:
Meaning it takes around 10 seconds when saving a file, where no generated code should be regenerated.
It appears that the runner is running quite quickly through the files, as they have no changes, but that still sums up to roughly 10 seconds.
Dart SDK Version:
2.18.6
What package(s) from this repo you are using, and the version:
2.1.11
What builder(s) you are using (or writing yourself). Try to give a short summary of what they do.
The text was updated successfully, but these errors were encountered: