Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework exclude rule merging #9307

Merged
merged 55 commits into from May 15, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
9814152
Rework exclude rule merging
eskatos Apr 27, 2019
2b35007
Remove indexed exclude factory
melix Apr 28, 2019
b33a6eb
Optimize `ExcludePair` for comparisons
melix Apr 29, 2019
67e1e79
Reuse caching at lower level
melix Apr 29, 2019
0d168e6
Optimize flattening
melix Apr 29, 2019
5e59d33
Use coarce grained locking
melix Apr 29, 2019
ee367ab
Remove redundant optimization
melix Apr 29, 2019
90fa836
Optimize 0 and 1 exclude metadata use cases
melix Apr 29, 2019
03fc570
Avoid the creation of intermediate datastructures when merging
melix Apr 30, 2019
1b3b8aa
Avoid computing the same exclusion filter multiple times
melix Apr 30, 2019
b8c8c34
Remove unused class
melix Apr 30, 2019
9f7d23f
Add union/intersection simplification
melix Apr 30, 2019
61b9c74
Give up a bit on immutability for the sake of performance
melix Apr 30, 2019
d8c428e
Fix composite exclude hashcode
melix Apr 30, 2019
b0288f0
Use hash set estimate size to avoid resizes
melix May 2, 2019
d7989ee
Perform simplification only if specs contain expected types
melix May 2, 2019
b72718e
Minor refactoring for readability
melix May 2, 2019
3fc2b2e
Avoid recomputing the same exclude filter multiple times
melix May 2, 2019
a329efa
Cache `EdgeState` and `DependencyState`
melix May 2, 2019
24d0c1b
De-duplicate dependencies before iterating
melix May 2, 2019
7cfb2ec
Cache edge exclusions for constraints
melix May 3, 2019
b948e0f
Avoid recomputing the dependency states
melix May 3, 2019
8d1a28c
Restore test combinations
melix May 3, 2019
9863bb0
Avoid adding the same dependency reasons multiple times
melix May 3, 2019
f0c56e2
Make selector state creation cheaper
melix May 3, 2019
93b024c
Small optimizations to make exclude computation faster
melix May 3, 2019
206d9a3
Fix merge conflict
melix May 3, 2019
c41d668
Reduce overhead of caching excludes
melix May 4, 2019
4ed59f1
Avoid de-duplicating dependencies
melix May 4, 2019
d5263cf
Optimize selection in case a variant provides the implicit capability
melix May 4, 2019
0ba07b5
Avoid re-creating selection reason
melix May 4, 2019
de69bef
Sort module selectors
melix May 5, 2019
bd90c10
Compute "hasTransitiveEdges" on the go
melix May 5, 2019
8384412
Minor refactoring for readability
melix May 5, 2019
1dd6b51
Add some missing `equals/hashCode`
melix May 6, 2019
50973ce
Optimize sorting of module selectors
melix May 6, 2019
067178d
Add module exclusion specialization for 1 incoming edge
melix May 6, 2019
84a6b68
Simplify `VersionParser`
melix May 6, 2019
b0ca0a6
Only consider "required" latest selector
melix May 6, 2019
e231c77
Make selector sorting faster
melix May 6, 2019
dbb2ca6
Order selectors on the fly
melix May 6, 2019
5fcbee8
Avoid the use of `Objects.hashCode`
melix May 6, 2019
b4d8d0c
Use `emptyList` instead of `ImmutableList.of()`
melix May 6, 2019
f702011
Use a list instead of a set for dependency reasons
melix May 6, 2019
fbf3905
Fix NPE: version of capability may be null
melix May 6, 2019
0f5a865
Fix duplicate rule descriptors
melix May 6, 2019
1cc7ed0
Use cached dependency states for additional constraints
melix May 6, 2019
be092aa
Store potentially later activated constraints
melix May 8, 2019
59d2b1f
Fix constraint not being reactivated
melix May 9, 2019
a54fceb
Optimize variant transform selection
melix May 10, 2019
a9ec4b4
Deduplicate attributes when writing result to disk
melix May 14, 2019
5a1ba04
Cache desugaring of attributes
melix May 14, 2019
6181cf1
Cache more than one query in component attribute matching
melix May 14, 2019
867de47
Optimize zip/tar visit
melix May 14, 2019
a6577d6
Always add constraints to potentially activating list
melix May 14, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Expand Up @@ -104,11 +104,12 @@ private void visitImpl(FileVisitor visitor, InputStream inputStream) throws IOEx
NoCloseTarInputStream tar = new NoCloseTarInputStream(inputStream);
TarEntry entry;
File expandedDir = getExpandedDir();
boolean isFirstTimeVisit = !expandedDir.exists();
while (!stopFlag.get() && (entry = tar.getNextEntry()) != null) {
if (entry.isDirectory()) {
visitor.visitDir(new DetailsImpl(resource, expandedDir, entry, tar, stopFlag, chmod));
visitor.visitDir(new DetailsImpl(resource, isFirstTimeVisit, expandedDir, entry, tar, stopFlag, chmod));
} else {
visitor.visitFile(new DetailsImpl(resource, expandedDir, entry, tar, stopFlag, chmod));
visitor.visitFile(new DetailsImpl(resource, isFirstTimeVisit, expandedDir, entry, tar, stopFlag, chmod));
}
}
}
Expand Down Expand Up @@ -189,13 +190,15 @@ private static class DetailsImpl extends AbstractFileTreeElement implements File
private final NoCloseTarInputStream tar;
private final AtomicBoolean stopFlag;
private final ReadableResourceInternal resource;
private final boolean isFirstTimeVisit;
private final File expandedDir;
private File file;
private boolean read;

public DetailsImpl(ReadableResourceInternal resource, File expandedDir, TarEntry entry, NoCloseTarInputStream tar, AtomicBoolean stopFlag, Chmod chmod) {
public DetailsImpl(ReadableResourceInternal resource, boolean isFirstTimeVisit, File expandedDir, TarEntry entry, NoCloseTarInputStream tar, AtomicBoolean stopFlag, Chmod chmod) {
super(chmod);
this.resource = resource;
this.isFirstTimeVisit = isFirstTimeVisit;
this.expandedDir = expandedDir;
this.entry = entry;
this.tar = tar;
Expand All @@ -213,7 +216,7 @@ public void stopVisiting() {
public File getFile() {
if (file == null) {
file = new File(expandedDir, entry.getName());
if (!file.exists()) {
if (isFirstTimeVisit) {
copyTo(file);
}
}
Expand Down
Expand Up @@ -79,6 +79,7 @@ public void visit(FileVisitor visitor) {
try {
ZipFile zip = new ZipFile(zipFile);
File expandedDir = getExpandedDir();
boolean isFirstTimeVisit = !expandedDir.exists();
try {
// The iteration order of zip.getEntries() is based on the hash of the zip entry. This isn't much use
// to us. So, collect the entries in a map and iterate over them in alphabetical order.
Expand All @@ -92,9 +93,9 @@ public void visit(FileVisitor visitor) {
while (!stopFlag.get() && sortedEntries.hasNext()) {
ZipEntry entry = sortedEntries.next();
if (entry.isDirectory()) {
visitor.visitDir(new DetailsImpl(zipFile, expandedDir, entry, zip, stopFlag, chmod));
visitor.visitDir(new DetailsImpl(zipFile, isFirstTimeVisit, expandedDir, entry, zip, stopFlag, chmod));
} else {
visitor.visitFile(new DetailsImpl(zipFile, expandedDir, entry, zip, stopFlag, chmod));
visitor.visitFile(new DetailsImpl(zipFile, isFirstTimeVisit, expandedDir, entry, zip, stopFlag, chmod));
}
}
} finally {
Expand All @@ -116,15 +117,17 @@ private File getExpandedDir() {

private static class DetailsImpl extends AbstractFileTreeElement implements FileVisitDetails {
private final File originalFile;
private final boolean isFirstTimeVisit;
private final File expandedDir;
private final ZipEntry entry;
private final ZipFile zip;
private final AtomicBoolean stopFlag;
private File file;

public DetailsImpl(File originalFile, File expandedDir, ZipEntry entry, ZipFile zip, AtomicBoolean stopFlag, Chmod chmod) {
public DetailsImpl(File originalFile, boolean isFirstTimeVisit, File expandedDir, ZipEntry entry, ZipFile zip, AtomicBoolean stopFlag, Chmod chmod) {
super(chmod);
this.originalFile = originalFile;
this.isFirstTimeVisit = isFirstTimeVisit;
this.expandedDir = expandedDir;
this.entry = entry;
this.zip = zip;
Expand All @@ -142,7 +145,7 @@ public void stopVisiting() {
public File getFile() {
if (file == null) {
file = new File(expandedDir, entry.getName());
if (!file.exists()) {
if (isFirstTimeVisit) {
copyTo(file);
}
}
Expand Down
Expand Up @@ -830,7 +830,7 @@ class DependencySubstitutionRulesIntegrationTest extends AbstractIntegrationSpec
resolve.expectGraph {
root(":impl", "depsub:impl:") {
module("org.utils:api:2.0")
edge("project :api", "org.utils:api:2.0").byConflictResolution("between versions 1.6 and 2.0").selectedByRule()
edge("project :api", "org.utils:api:2.0").byConflictResolution("between versions 2.0 and 1.6").selectedByRule()
}
}
}
Expand Down Expand Up @@ -952,7 +952,7 @@ class DependencySubstitutionRulesIntegrationTest extends AbstractIntegrationSpec
module("org.utils:b:1.3") {
module("org.utils:a:1.3")
}
edge("org.utils:a:1.2", "org.utils:a:1.3").byConflictResolution("between versions 1.2.1 and 1.3")
edge("org.utils:a:1.2", "org.utils:a:1.3").byConflictResolution("between versions 1.3 and 1.2.1")
}
}
}
Expand Down Expand Up @@ -1278,7 +1278,7 @@ Required by:
resolve.expectGraph {
root(":", ":depsub:") {
edge("org:a:1.0", "org:a:2.0") {
byConflictResolution("between versions 1.0 and 2.0")
byConflictResolution("between versions 2.0 and 1.0")
module("org:c:1.0")
}
edge("foo:b:1.0", "org:b:1.0") {
Expand Down Expand Up @@ -1318,7 +1318,7 @@ Required by:
resolve.expectGraph {
root(":", ":depsub:") {
edge("org:a:1.0", "org:a:2.0") {
byConflictResolution("between versions 1.0 and 2.0")
byConflictResolution("between versions 2.0 and 1.0")
module("org:c:1.0")
}
edge("foo:bar:baz", "org:b:1.0") {
Expand Down Expand Up @@ -1377,7 +1377,7 @@ Required by:
resolve.expectGraph {
root(":", ":depsub:") {
edge("org:a:1.0", "org:c:2.0") {
byConflictResolution("between versions 1.1 and 2.0")
byConflictResolution("between versions 2.0 and 1.1")
}
module("org:a:2.0") {
module("org:b:2.0") {
Expand Down Expand Up @@ -1449,7 +1449,7 @@ Required by:
resolve.expectGraph {
root(":", ":depsub:") {
edge("org:a:1.0", "org:a:2.0") {
byConflictResolution("between versions 1.0 and 2.0")
byConflictResolution("between versions 2.0 and 1.0")
module("org:c:1.0")
}
edge("foo:bar:baz", "org:b:1.0") {
Expand Down
Expand Up @@ -16,9 +16,11 @@

package org.gradle.integtests.resolve


import org.gradle.integtests.fixtures.executer.GradleContextualExecuter
import spock.lang.IgnoreIf
import spock.lang.Issue
import spock.lang.Unroll

/**
* Demonstrates the resolution of dependency excludes in published module metadata.
Expand Down Expand Up @@ -109,7 +111,8 @@ task check(type: Sync) {
*
* Exclude is applied to dependency a->b
*/
def "dependency exclude for group or module applies to child module of dependency"() {
@Unroll
def "dependency exclude for group or module applies to child module of dependency (#excluded)"() {
given:
def expectResolved = ['a', 'b', 'c', 'd', 'e'] - expectExcluded
repository {
Expand Down Expand Up @@ -202,7 +205,8 @@ task check(type: Sync) {
*
* Selective exclusions are applied to dependency a->b
*/
def "can exclude transitive dependencies"() {
@Unroll
def "can exclude transitive dependencies (#condition)"() {
repository {
'a:a:1.0' {
dependsOn group: 'b', artifact: 'b', version: '1.0', exclusions: excludes
Expand Down
Expand Up @@ -156,7 +156,7 @@ class PublishedDependencyConstraintsIntegrationTest extends AbstractModuleDepend
}
}
if (available) {
edge("org:foo:1.0","org:foo:1.1").byConflictResolution("between versions 1.0 and 1.1")
edge("org:foo:1.0","org:foo:1.1").byConflictResolution("between versions 1.1 and 1.0")
} else {
module("org:foo:1.0")
}
Expand Down Expand Up @@ -221,7 +221,7 @@ class PublishedDependencyConstraintsIntegrationTest extends AbstractModuleDepend
}
module("org:first-level2:1.0") {
if (available) {
edge("org:foo:1.0","org:foo:1.1").byConflictResolution("between versions 1.0 and 1.1")
edge("org:foo:1.0","org:foo:1.1").byConflictResolution("between versions 1.1 and 1.0")
} else {
module("org:foo:1.0")
}
Expand Down Expand Up @@ -395,7 +395,7 @@ class PublishedDependencyConstraintsIntegrationTest extends AbstractModuleDepend
module("org:first-level:1.0") {
if (available) {
constraint("org:bar:1.1", "org:foo:1.1").selectedByRule()
edge("org:foo:1.0", "org:foo:1.1").byConflictResolution("between versions 1.0 and 1.1")
edge("org:foo:1.0", "org:foo:1.1").byConflictResolution("between versions 1.1 and 1.0")
} else {
module("org:foo:1.0")
}
Expand Down
Expand Up @@ -398,7 +398,7 @@ class ResolveConfigurationDependenciesBuildOperationIntegrationTest extends Abst
op.details.configurationName == "compile"
op.failure == "org.gradle.api.artifacts.ResolveException: Could not resolve all dependencies for configuration ':compile'."
failure.assertHasCause("""Conflict(s) found for the following module(s):
- org:leaf between versions 1.0 and 2.0""")
- org:leaf between versions 2.0 and 1.0""")
op.result != null
op.result.resolvedDependenciesCount == 2
}
Expand Down
Expand Up @@ -601,7 +601,7 @@ class RichVersionConstraintsIntegrationTest extends AbstractModuleDependencyReso
Dependency path ':test:unspecified' --> 'org:foo:{strictly [0,1]}'""")
}

void "should pass if strict version ranges overlap"() {
void "should pass if strict version ranges overlap"() {
given:
repository {
'org:foo' {
Expand Down
Expand Up @@ -160,11 +160,11 @@ project(':tool') {
root(":tool", "test:tool:") {
project(":api", "test:api:") {
configuration = "runtimeElements"
edge("org:foo:1.3.3", "org:foo:1.4.4").byConflictResolution("between versions 1.3.3 and 1.4.4")
edge("org:foo:1.3.3", "org:foo:1.4.4").byConflictResolution("between versions 1.4.4 and 1.3.3")
}
project(":impl", "test:impl:") {
configuration = "runtimeElements"
module("org:foo:1.4.4").byConflictResolution("between versions 1.3.3 and 1.4.4")
module("org:foo:1.4.4").byConflictResolution("between versions 1.4.4 and 1.3.3")
}
}
}
Expand Down Expand Up @@ -210,10 +210,10 @@ task resolve {
resolve.expectGraph {
root(":", "org:test:1.0") {
module("org:bar:1.0") {
edge("org:foo:1.3.3", "org:foo:1.4.4").byConflictResolution("between versions 1.3.3 and 1.4.4")
edge("org:foo:1.3.3", "org:foo:1.4.4").byConflictResolution("between versions 1.4.4 and 1.3.3")
}
module("org:baz:1.0") {
module("org:foo:1.4.4").byConflictResolution("between versions 1.3.3 and 1.4.4")
module("org:foo:1.4.4").byConflictResolution("between versions 1.4.4 and 1.3.3")
}
}
}
Expand Down Expand Up @@ -262,7 +262,7 @@ dependencies {
}
module("org:two:1.0") {
module("org:dep-2.0-bringer:1.0") {
edge("org:dep:2.0", "org:dep:2.5").byConflictResolution("between versions 2.0 and 2.5")
edge("org:dep:2.0", "org:dep:2.5").byConflictResolution("between versions 2.5 and 2.0")
}
module("org:control-1.2-bringer:1.0") {
module("org:control:1.2")
Expand Down
Expand Up @@ -107,7 +107,7 @@ class AlignmentIntegrationTest extends AbstractAlignmentSpec {
module('org:core:1.1')
}
module("outside:module:1.0") {
edge('org:core:1.0', 'org:core:1.1').byConflictResolution("between versions 1.0 and 1.1")
edge('org:core:1.0', 'org:core:1.1').byConflictResolution("between versions 1.1 and 1.0")
}
}
}
Expand Down Expand Up @@ -187,7 +187,7 @@ class AlignmentIntegrationTest extends AbstractAlignmentSpec {
root(":", ":test:") {
module("org:xml:1.0") {
edge('org:core:1.0', 'org:core:1.1')
.byConflictResolution("between versions 1.0 and 1.1")
.byConflictResolution("between versions 1.1 and 1.0")
.byConstraint("belongs to platform org:platform:1.1")
}
module("org:json:1.1") {
Expand Down Expand Up @@ -529,7 +529,7 @@ class AlignmentIntegrationTest extends AbstractAlignmentSpec {
}
module('org2:foo:1.0') {
edge('org4:a:1.0', 'org4:a:1.1') {
byConflictResolution("between versions 1.0 and 1.1")
byConflictResolution("between versions 1.1 and 1.0")
}
}
module('org3:bar:1.0') {
Expand Down Expand Up @@ -596,7 +596,7 @@ class AlignmentIntegrationTest extends AbstractAlignmentSpec {
byConstraint("belongs to platform org:platform:1.1")
// byReason("version 1.1 is buggy") // TODO CC: uncomment when we collect rejection from component selection rule
edge('org:core:1.0', 'org:core:1.1') {
byConflictResolution("between versions 1.0 and 1.1")
byConflictResolution("between versions 1.1 and 1.0")
}
}
module("org:json:1.1") {
Expand Down Expand Up @@ -731,7 +731,7 @@ class AlignmentIntegrationTest extends AbstractAlignmentSpec {
module("org.apache.groovy:core:2.5")
}
edge("org.springframework:core:1.0", "org.springframework:core:1.1") {
byConflictResolution("between versions 1.0 and 1.1")
byConflictResolution("between versions 1.1 and 1.0")
}
}
}
Expand Down Expand Up @@ -813,7 +813,7 @@ class AlignmentIntegrationTest extends AbstractAlignmentSpec {
module("org.apache.groovy:core:2.5")
}
edge("org.springframework:core:1.0", "org.springframework:core:1.1") {
byConflictResolution("between versions 1.0 and 1.1")
byConflictResolution("between versions 1.1 and 1.0")
}
}
}
Expand Down
Expand Up @@ -78,7 +78,7 @@ class ResolutionResultApiIntegrationTest extends AbstractDependencyResolutionTes
then:
output.contains """
cool-project:5.0 root
foo:1.0 between versions 0.5 and 1.0
foo:1.0 between versions 1.0 and 0.5
leaf:2.0 forced
bar:1.0 requested
baz:1.0 requested
Expand Down Expand Up @@ -129,8 +129,8 @@ baz:1.0 requested
descriptions.each {
println "\$it.cause : \$it.description"
}
def descriptor = descriptions.find { it.cause == ComponentSelectionCause.REQUESTED }
assert descriptor?.description == 'second reason'
def descriptors = descriptions.findAll { it.cause == ComponentSelectionCause.REQUESTED }
assert descriptors.description == ['first reason', 'second reason']
}
}
}
Expand Down Expand Up @@ -200,8 +200,8 @@ baz:1.0 requested
descriptions.each {
println "\$it.cause : \$it.description"
}
def descriptor = descriptions.find { it.cause == ComponentSelectionCause.REQUESTED }
assert descriptor?.description == 'second reason'
def descriptors = descriptions.findAll { it.cause == ComponentSelectionCause.REQUESTED }
assert descriptors.description == ['requested', 'second reason']
}
}
}
Expand All @@ -217,14 +217,14 @@ baz:1.0 requested
root(":", ":test:") {
module('org.test:a:1.0:runtime') {
edge('org.test:leaf:0.9', 'org.test:leaf:1.1')
.byConflictResolution("between versions 1.0 and 1.1") // conflict with the version requested by 'b'
.byConflictResolution("between versions 1.1 and 1.0") // conflict with the version requested by 'b'
.byReason('second reason') // this comes from 'b'
.selectedByRule("substitute 0.9 with 1.0")
}
module('org.test:b:1.0:runtime') {
module('org.test:leaf:1.1')
.selectedByRule("substitute 0.9 with 1.0")
.byConflictResolution("between versions 1.0 and 1.1")
.byConflictResolution("between versions 1.1 and 1.0")
.byReason('second reason')
}
}
Expand Down
Expand Up @@ -355,7 +355,7 @@ class MultipleVariantSelectionIntegrationTest extends AbstractModuleDependencyRe
resolve.expectGraph {
root(":", ":test:") {
edge('org:foo:1.0', 'org:foo:1.1') {
byConflictResolution('between versions 1.0 and 1.1')
byConflictResolution('between versions 1.1 and 1.0')
// the following assertion is true but limitations to the test fixtures make it hard to check
//variant('altruntime', [custom: 'c3', 'org.gradle.status': defaultStatus()])
variant('runtime', [custom: 'c2', 'org.gradle.status': defaultStatus(), 'org.gradle.usage': 'java-runtime-jars'])
Expand Down Expand Up @@ -586,7 +586,7 @@ class MultipleVariantSelectionIntegrationTest extends AbstractModuleDependencyRe
resolve.expectGraph {
root(":", ":test:") {
edge('org:foo:1.0', 'org:foo:1.1') {
byConflictResolution('between versions 1.0 and 1.1')
byConflictResolution('between versions 1.1 and 1.0')
variant('runtime', [custom: 'c2', 'org.gradle.status': defaultStatus(), 'org.gradle.usage': 'java-runtime-jars'])
artifact group: 'org', module: 'foo', version: '1.0', classifier: 'c2'
}
Expand Down
Expand Up @@ -91,7 +91,7 @@ class DependencyConstraintsAndResolutionStrategiesIntegrationTest extends Abstra

then:
failure.assertHasCause """Conflict(s) found for the following module(s):
- org:foo between versions 1.0 and 1.1"""
- org:foo between versions 1.1 and 1.0"""
}

void "dependency substitution rules are applied to dependency constraints"() {
Expand Down