Graph serialization error with projects being replaced by binary dependencies #13329
Comments
|
Thanks for the detailed report. There are a couple of issues here, which I'll address separately. Binary dependency is replaced by resolving projectFirst, when a configuration is declared for a project, the project module itself is always going to be the root node of the resolved configuration. So if project The configuration X does not represent a module, doesn't have its own coordinates, and thus cannot exist independently in the graph. There's currently no way to use a module other than the resolving project as the resolution root. Second, when 2 versions of the same module are present, Gradle chooses the newest one via conflict resolution.
Error:
|
|
Thanks @bigdaz! I understand from your response that the project is conceptually the root of the graph for practical reasons, which means that it needs to participate in the resolution. I don't think that's a problem in itself – the problem is that it doesn't look like there's a way to configure conflict resolution when the project is involved. The existing policies (latest module or prefer projects) can't be configured to select the binary dependency. Also, the conflict doesn't seem to be considered as such, as reflected in the absence of a conflict resolution message. I addressed this problem by removing the project version, and setting the version directly in the publication itself. Still, it would be useful for the behavior to be documented (I don't think it is?), and the response to it to be more consistent. As for the issue with the binary store, I can repro consistently, and I'll try to set up a small reproducer. As far as I can tell, the graph resolution is dropping a bunch of valid nodes for no clear reason, but it's still emitting edges that point at the missing nodes. |
|
Ah, it seems that the issue is caused by a combination of project and binary dependencies using BOM files, where some of the conflicts between project and binary dependencies are left unresolved. At the time of the graph resolution, the orphan nodes cause the graph traversal to be incomplete, dropping edges and leaving orphan nodes behind. The missing dependencies cause the build to fail, and the invalid graph causes the build scan to fail. I built a reproducer from my original project and shared with @bigdaz and @jjohannes. |
|
Thanks @cesar1000. I'm not going to have capacity to investigate the Binary store issue. Could you please also share your reproducer project with @ljacomet and @melix? |
|
Done! Thanks for your direction yesterday, @bigdaz. |
|
Awesome, thanks, Louis! Coincidentally, I just found another issue that repros in the same repo after a minor change. I think it's unrelated, so filed separately as #13353. |
|
I have figured out the issue for the graph serialization issue. However I have to admit this graph was really strange until I figured it out 😉 The initial report about the spurious conflict is real. Because the projects have no version defined, the moment a binary dependency with the same coordinates but a version enters the graph, it wins and pretty much replaces all the projects with the binary versions. An alternative is to use The root cause of that behaviour is that the None of that is unexpected behavior, so once the binary store issue is fixed and the version conflict clarified, there should not be another change. I'll follow up once I have a nightly that can be tested on the reproducer. |
|
Yeah, it's an unusual graph! This is a layered annotation processor that runs on itself, so I need the processor classpaths to use a binary dependency, and the runtime classpath to use the project dependency, when available, and the binary dependency otherwise. I worked around by using The behavior that I saw, however, looked unexpected: the conflicting dependencies were dropped from the configuration altogether, not just resolved to unexpected dependencies. The resulting dependency graph did not match up edges and nodes correctly, which resulted in nodes being dropped even if referenced by some of the edges (which ultimately caused the crash). This would should up in the build as compilation failures with symbols not in the classpath. Is that what you're addressing in your fix? |
Before this change, some node that carried constraints would remain registered as interested in pending dependencies. This could cause invalid deferred selection. With this set of changes, such state is cleaned up when a node that carries constraints no longer participates in the graph, either because it is not transitive or because it is no longer selected. Issue #13329
|
I'm seeing an issue in this project where dependency resolution hangs in what seems to be an infinite loop (following a cycle in the graph?). I couldn't figure out how to isolate a repro. Before I put in any more time, is there a chance that this fix may address this kind of issue? |
|
Hey @cesar1000, You can try with the following nightly: |
ljacomet
changed the title
|
Thanks! The nightly does fix the original issue. I does not fix the dependency resolution hanging in an infinite loop though – would you want me to add a separate ticket for that? What kind of information would help troubleshoot? If it helps, here's a dump of a full cycle in the infinite loop of dependency resolution: |
|
Great that the initial issue is confirmed fixed. Please do open a new bug for that infinite loop issue, providing as much information as you can. A couple thread dumps could help as well. |
Before this change, some node that carried constraints would remain registered as interested in pending dependencies. This could cause invalid deferred selection. With this set of changes, such state is cleaned up when a node that carries constraints no longer participates in the graph, either because it is not transitive or because it is no longer selected. Issue #13329
|
Closing as the fix for the underlying graph serialization problem has been fixed and merged. |
|
Awesome, thanks! Reported #13450. |
Before this change, some node that carried constraints would remain registered as interested in pending dependencies. This could cause invalid deferred selection. With this set of changes, such state is cleaned up when a node that carries constraints no longer participates in the graph, either because it is not transitive or because it is no longer selected. Issue #13329
I am developing a annotation processor that runs on its own project, using a binary dependency. In order to avoid conflicts between project dependencies and binary dependencies, I'm curating my configurations carefully to isolate binary dependencies from the corresponding project dependencies. Dependency resolution, however, selects the a project dependency even if it's not present in my configuration.
Expected Behavior
The module resolution mechanism in a configuration should only draw candidates from the configuration itself. In particular, project dependencies that haven't been added to the configuration explicitly should not affect the resolution.
Current Behavior
In the current behavior, the project is added to the dependency resolution as an automatic candidate (in
SelectorStateResolver, it seems), with no ability to configure. I put up a small repro of this issue here. In this example, theprocessorconfiguration contains v0.0.1 of the binary artifact. If the version of the project is set to v0.0.0, the configuration resolves the artifact to v0.0.1, but reports about a spurious conflict with v0.0.0:If setting the project version to v0.0.2, the resolution selects the project, and does not report a conflict:
Context
In more complex projects, this seems to lead to downstream issues where the resolved project dependencies are dropped from their configurations entirely. Also, the binary store of dependencies gets corrupted in this configuration, which causes both the
dependenciestask and the build scan to fail with exceptions like the following:These issues are more involved to repro, but I can try to put something together if it helps.
Steps to Reproduce
https://github.com/cesar1000/gradle-recursive-dependency
Your Environment
I'm using Gradle v6.5.
Build scan URL:
With v0.0.0: https://scans.gradle.com/s/sba5ra5cxfcsy/dependencies?focusedDependency=WzAsMCwwLFswLDAsWzBdXV0&toggled=W1swXSxbMCwwXV0
With v0.0.2: https://scans.gradle.com/s/22nsrpchulvxw/dependencies?focusedDependency=WzAsMCwwLFswLDAsWzBdXV0&toggled=W1swXSxbMCwwXV0
The text was updated successfully, but these errors were encountered: