Uploaded image for project: 'Maven Resolver'
  1. Maven Resolver
  2. MRESOLVER-133

Improve resolver performance by using breadth-first search

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.4.2
    • 1.8.0
    • Resolver
    • None

    Description

       

      I believe the maven resolver is unnecessarily inefficient because it performs a depth-first search of components to resolve dependencies. Consider the case when dependencies use version ranges, the user intent is for maven to resolve with the highest versions of dependencies that satisfy all constraints. If maven were to use a breadth-first search (and terminate searching as soon as a solution is found)  then in many cases a valid set of dependencies can be resolved (at the top of the version ranges) without requiring that all historical versions are resolvable. One should get the same answer with both depth-first and breadth first strategies, but with the breadth-first approach not being vulnerable to a missing parent POM somewhere in history making it impossible to build the head of code. Additionally, I suspect that breadth-first would be faster and use less memory than depth first.

       

      Additionally the depth-first approach has a weakness that when ny version of a parent pom of a component referenced in a dependency tree of another component is missing maven fails to resolve dependencies. One get a message of the form:

      Failed to execute goal on project module: Could not resolve dependencies for project baddepdemo.project2:module:jar:1: Failed to collect dependencies ...

       

      Currently the only way to resolve this issue is one of three ways:

       1) restore a missing parent POM into the repository history, or

       2) delete all modules  associated with the missing parent POM from the repository

       3) manually adjust version ranges in consumer dependencies to exclude the bad versions of dependencies that refer to the missing parent POM.

       

      What I would like is a configuration switch that would allow one to select between the two search strategies So that the manual interventions described above are not required.

       

      I have include a zip file that include the minimal projects needed to demonstrate the dependency resolution problem:

      project 1 has a module and parent pom.

      project 2 is a single pom that has a dependency on the module in project 1. Project 2 uses a dependency range [1,) that indicates that the latest version of project1's module is to be used.

      If one builds two versions (1 and 2) of project 1, project2 will resolve to use version 2 as expected. However if you delete the parent pom of  project1 from the repository maven cannot resolve dependencies and fails. If the version range in project 2 is changed to [2,) then the expected behavior is observed.

      The zip file contains a shell script (demo.sh) that can be run without parameters to demonstrate the behavior when all versions are present in the repository. Run it with 1 as a parameter (the lower end of the version range used in project2) and the script will delete the parent pom from project 1 and the error condition will be demonstrated.  Run it with 2 and maven will resolve dependencies as version1 of project1 is explicitly excluded from the dependency resolution process.

       

      I am also willing to look at the source and propose a patch, but I would need guidance on which modules/source I should look at.

       

       

      Attachments

        1. mvnbaddeps.zip
          6 kB
          Gregory Ducharme

        Issue Links

          Activity

            michael-o Michael Osipov added a comment -

            I ran your code with 3.5.4 and 3.7.0-SNAPSHOT. I had BUILD SUCCESS everytime. Please provide logs.

            michael-o Michael Osipov added a comment - I ran your code with 3.5.4 and 3.7.0-SNAPSHOT. I had BUILD SUCCESS everytime. Please provide logs.

            michael-o

            Did you run the test cases as specified?

            1. demo.sh  — (with no parameters) should be success
            2. demo.sh  1   ---- should fail
            3. demo.sh 2  — should succeed

             

             

            cannucklehead Gregory Ducharme added a comment - michael-o Did you run the test cases as specified? demo.sh  — (with no parameters) should be success demo.sh  1   ---- should fail demo.sh 2  — should succeed    
            michael-o Michael Osipov added a comment -

            No, I did not. Next time please write explicit commands. Going through...

            michael-o Michael Osipov added a comment - No, I did not. Next time please write explicit commands. Going through...
            michael-o Michael Osipov added a comment -

            First of all, this belongs to Resolver, issue moved. Have a look at the exception:

            Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at baddepdemo.project1:module:jar:1
                at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.collectDependencies (DefaultDependencyCollector.java:288)
                at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies (DefaultRepositorySystem.java:284)
            

            How is this supposed to work? Resolver sees your range request. To solve this, it requires maven-metadata.xml to find the limited upper range bound. Just because 2 is installed, it does not mean it is the best available version. It may exist version 3 or newer remotely.

            In my opinion, this works as designed.

            khmarbaise, WDYT?

            michael-o Michael Osipov added a comment - First of all, this belongs to Resolver, issue moved. Have a look at the exception: Caused by: org.eclipse.aether.collection.DependencyCollectionException: Failed to collect dependencies at baddepdemo.project1:module:jar:1 at org.eclipse.aether.internal.impl.collect.DefaultDependencyCollector.collectDependencies (DefaultDependencyCollector.java:288) at org.eclipse.aether.internal.impl.DefaultRepositorySystem.collectDependencies (DefaultRepositorySystem.java:284) How is this supposed to work? Resolver sees your range request. To solve this, it requires maven-metadata.xml to find the limited upper range bound. Just because 2 is installed, it does not mean it is the best available version. It may exist version 3 or newer remotely. In my opinion, this works as designed. khmarbaise , WDYT?

            Hello Michael:

             

            I am glad you were able to reproduce the issue and route it the Resolver. Thank-you.

             

            However, my concern is not that the resolver looks for higher versions. This is expected.

            My concern is that the resolver looks for (and errors out) on looking for lower versions (version 1) that are not part of the resolved version set (test case 2). 

            Note that the resolved versions for test case 1 and 3 are the same. I would expect that test case 2 would yield the same solution as well.

             

            The version ranges in the POMs should be interpreted as 'resolve to the highest versions of the dependencies'. Is it reasonable for the resolver to fail if an old (lower) version that is not part of (nor affects) the solution set is missing?

             

            cannucklehead Gregory Ducharme added a comment - Hello Michael:   I am glad you were able to reproduce the issue and route it the Resolver. Thank-you.   However, my concern is not that the resolver looks for higher versions. This is expected. My concern is that the resolver looks for (and errors out) on looking for lower versions (version 1) that are not part of the resolved version set (test case 2).  Note that the resolved versions for test case 1 and 3 are the same. I would expect that test case 2 would yield the same solution as well.   The version ranges in the POMs should be interpreted as 'resolve to the highest versions of the dependencies'. Is it reasonable for the resolver to fail if an old (lower) version that is not part of (nor affects) the solution set is missing?  
            michael-o Michael Osipov added a comment -

            Let me evaluate that...

            michael-o Michael Osipov added a comment - Let me evaluate that...
            michael-o Michael Osipov added a comment - - edited

            OK, I went through. Case 1 and 3 are not identical: The second one does delete version 1. Please look closely at case 2: demo.sh 1. Module-1, v1 requires parent, v1 which is not present. I assume your stance is that: If from maven-metadata.xml a higher version is requested from the range, don't download the lower bound of the range because a newer one is available. So it should go straight to version 2 w/o even touching version 1 at all.

            If so, I don't consider that as a bug, but merely as weakness because nothing is broken. I recommend setting a breakpoint in org.apache.maven.repository.internal.DefaultVersionRangeResolver.resolveVersionRange(RepositorySystemSession, VersionRangeRequest) and try to analyze this. I am quite certain that when versions are resolved, it tries to read all metadata in org.apache.maven.repository.internal.DefaultVersionRangeResolver.readVersions(RepositorySystemSession, RequestTrace, Metadata, ArtifactRepository, VersionRangeResult).

            michael-o Michael Osipov added a comment - - edited OK, I went through. Case 1 and 3 are not identical: The second one does delete version 1. Please look closely at case 2: demo.sh 1 . Module-1, v1 requires parent, v1 which is not present. I assume your stance is that: If from maven-metadata.xml a higher version is requested from the range, don't download the lower bound of the range because a newer one is available. So it should go straight to version 2 w/o even touching version 1 at all. If so, I don't consider that as a bug, but merely as weakness because nothing is broken. I recommend setting a breakpoint in org.apache.maven.repository.internal.DefaultVersionRangeResolver.resolveVersionRange(RepositorySystemSession, VersionRangeRequest) and try to analyze this. I am quite certain that when versions are resolved, it tries to read all metadata in org.apache.maven.repository.internal.DefaultVersionRangeResolver.readVersions(RepositorySystemSession, RequestTrace, Metadata, ArtifactRepository, VersionRangeResult) .
            michael-o Michael Osipov added a comment -

            So I stepped through the code. It does what I have assumed. It goes through all available versions, resolves dependencies and parent and then makes a decision for a version. This is the behavior you see.

            michael-o Michael Osipov added a comment - So I stepped through the code. It does what I have assumed. It goes through all available versions, resolves dependencies and parent and then makes a decision for a version. This is the behavior you see.

            I agree that is how maven works. The test cases were crafted to show the 'weakness' as you call it.

            If maven used a breadth first search, only downloading lower versions as needed until a solution is resolved. The resolution should be faster (fewer GET requests to the repository manager) and more resilient in the face of missing older versions.

             

            So maybe this should be logged as an enhancement. 

             

            I am quite happy to look in the code, make some changes and propose a PR...

             

            cannucklehead Gregory Ducharme added a comment - I agree that is how maven works. The test cases were crafted to show the 'weakness' as you call it. If maven used a breadth first search, only downloading lower versions as needed until a solution is resolved. The resolution should be faster (fewer GET requests to the repository manager) and more resilient in the face of missing older versions.   So maybe this should be logged as an enhancement.    I am quite happy to look in the code, make some changes and propose a PR...  
            michael-o Michael Osipov added a comment -

            I agree that it is much more efficient to download what is really reaquired. A breadth-first seach would be helpful here. Please change title and description of this issue to reflect the improvement you are trying to achieve because w/o it is not clear what problem you are trying to solve.

            michael-o Michael Osipov added a comment - I agree that it is much more efficient to download what is really reaquired. A breadth-first seach would be helpful here. Please change title and description of this issue to reflect the improvement you are trying to achieve because w/o it is not clear what problem you are trying to solve.

            Updated the title, type and description as requested.

            cannucklehead Gregory Ducharme added a comment - Updated the title, type and description as requested.
            michael-o Michael Osipov added a comment -

            OK, makes sense now. If you want to work on a PR, please go ahead, but I think this is going to be tricky w/o breaking (any) edge cases.

            michael-o Michael Osipov added a comment - OK, makes sense now. If you want to work on a PR, please go ahead, but I think this is going to be tricky w/o breaking (any) edge cases.
            michael-o Michael Osipov added a comment -

            wecai, can you take a look at this and how this intersects with your issue/PR?

            michael-o Michael Osipov added a comment - wecai , can you take a look at this and how this intersects with your issue/PR?
            wecai wei cai added a comment - - edited

            michael-o 

            I tested the demo.sh 1 with the PR in MRESOLVER-228 enabled, it also fail, there is no behavior change.
            This means PR of MRESOLVER-228 won't fix the issue described in this ticket.

            As below code shows, there is no code change for "resolveVersionRange" part.

            https://github.com/apache/maven-resolver/blob/78e5e98b6430ba71c982cbc9bce6f603cc2cb286/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/DefaultDependencyCollector.java#L200

            Before I submitted the PR in MRESOLVER-228, I actually noticed the Jira and implemented a breadth-first approach, we did not find any memory reduction with BFS solution when running some of apps with complex dependencies .

            Maven is proud of its "nearest (to root node) wins" strategy when addressing dependency version conflicts, so maven has to:

            • resolve all tree paths
            • then resolve conflicts of all paths to pick up dependency version when there are version conflicts

            Even with BFS solution, maven still need to resolve all tree paths,  so same memory cost and same resolve time there.  This is why I decided to go with a Skip & Reconcile approach as it can skipped lots of unnecessary calculation.

             

            wecai wei cai added a comment - - edited michael-o   I tested the demo.sh 1  with the PR in MRESOLVER-228 enabled, it also fail, there is no behavior change. This means PR of MRESOLVER-228 won't fix the issue described in this ticket. As below code shows, there is no code change for "resolveVersionRange" part. https://github.com/apache/maven-resolver/blob/78e5e98b6430ba71c982cbc9bce6f603cc2cb286/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/DefaultDependencyCollector.java#L200 Before I submitted the PR in MRESOLVER-228 , I actually noticed the Jira and implemented a breadth-first approach, we did not find any memory reduction with BFS solution when running some of apps with complex dependencies . Maven is proud of its "nearest (to root node) wins" strategy when addressing dependency version conflicts, so maven has to: resolve all tree paths then resolve conflicts of all paths to pick up dependency version when there are version conflicts Even with BFS solution, maven still need to resolve all tree paths,  so same memory cost and same resolve time there.  This is why I decided to go with a Skip & Reconcile approach as it can skipped lots of unnecessary calculation.  

            Hello folks:

             

            I would like to clarify the use case for the performance aspect of this issue:

            Define a component with  a large number of dependencies (try 100) with version ranges like [a.b.c,) or [a.b.c]

            Give them in turn a large number of dependencies of the same form. 

            Just for for fun do the same for transitive dependencies.

            And to finish the scenario give each artifact 100 versions .

             

            Think in terms of trunk based development. You only want to deploy the latest/greatest components to your application.

             

            So the tree of all possible combinations will be huge. Maven will download all the poms and we see resolution times of 30 minutes. The calculation time in the resolver code is not the issue. The downloading of all those poms is the time consumer. 

             

            I would expect that a breadth-first solution that terminates upon finding the first tree path that resolves to a solution would do so long before all poms were downloaded and all alternatives explored.  But maybe I am wrong. 

             

            But we have some work arounds:

            1) in some cases, we don't use maven at all, we write our own dependency resolution mechanism using groovy and the maven version comparison jar.

            2) refactor the code to reduce inter-dependencies between components.

            3) we delete old content from our repositories. Keep the latest 10.

            4) we also create bill of material poms that we update daily to limit the version ranges so that the lowest range is a version built last week. 

             

            Work-around 3 is problematic in that some cases, we end up with a mismatch of the number of versions of a child module compared to the number of versions of a parent module. This leads to the maven failure described in the original use case and demonstrated in the code supplied as the test case to this issue. 

            Work-around 4 is problematic as it turns a number of independent components into a single monolithic 'build', breaking work-around 2.

             

            Thank-you for your efforts on this issue.

            cannucklehead Gregory Ducharme added a comment - Hello folks:   I would like to clarify the use case for the performance aspect of this issue: Define a component with  a large number of dependencies (try 100) with version ranges like [a.b.c,) or [a.b.c] Give them in turn a large number of dependencies of the same form.  Just for for fun do the same for transitive dependencies. And to finish the scenario give each artifact 100 versions .   Think in terms of trunk based development. You only want to deploy the latest/greatest components to your application.   So the tree of all possible combinations will be huge. Maven will download all the poms and we see resolution times of 30 minutes. The calculation time in the resolver code is not the issue. The downloading of all those poms is the time consumer.    I would expect that a breadth-first solution that terminates upon finding the first tree path that resolves to a solution would do so long before all poms were downloaded and all alternatives explored.  But maybe I am wrong.    But we have some work arounds: 1) in some cases, we don't use maven at all, we write our own dependency resolution mechanism using groovy and the maven version comparison jar. 2) refactor the code to reduce inter-dependencies between components. 3) we delete old content from our repositories. Keep the latest 10. 4) we also create bill of material poms that we update daily to limit the version ranges so that the lowest range is a version built last week.    Work-around 3 is problematic in that some cases, we end up with a mismatch of the number of versions of a child module compared to the number of versions of a parent module. This leads to the maven failure described in the original use case and demonstrated in the code supplied as the test case to this issue.  Work-around 4 is problematic as it turns a number of independent components into a single monolithic 'build', breaking work-around 2.   Thank-you for your efforts on this issue.
            ibabiankou Ivan Babiankou added a comment -

            cannucklehead I've been looking into this ticket, after michael-o mentioned it in my PR and if I understand both the problem and proposed improvement correctly it can be fairly easily implemented:

            This loop should iterate over reversed list of versions, it should only try to get artifact descriptor and break as soon as got one successfully. If no versions are available, then the build could be failed with appropriate message.

            The dependency resolution then should be only done for the selected version only.
            WDYT? Does it make sense to you guys?

            ibabiankou Ivan Babiankou added a comment - cannucklehead I've been looking into this ticket, after michael-o mentioned it in my PR and if I understand both the problem and proposed improvement correctly it can be fairly easily implemented: This loop should iterate over reversed list of versions, it should only try to get artifact descriptor and break as soon as got one successfully. If no versions are available, then the build could be failed with appropriate message. The dependency resolution then should be only done for the selected version only. WDYT? Does it make sense to you guys?
            wecai wei cai added a comment -

            ibabiankou michael-o 
            Here maven is using below code to resolve only the latest version
            Version version = versions.get( versions.size() - 1 );
            https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/DefaultDependencyCollector.java#L187

            And here maven is resolving all versions of a version range:
            for ( Version version : versions )
            https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/DefaultDependencyCollector.java#L405

            Why not keep the same?
            We don't need to iterate over reversed list of versions, I think resolve the latest version by "Version version = versions.get( versions.size() - 1 )" is enough.
            Any thoughts?

            With such change, no tests broken at all.

            wecai wei cai added a comment - ibabiankou michael-o   Here maven is using below code to resolve only the latest version Version version = versions.get( versions.size() - 1 ); https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/DefaultDependencyCollector.java#L187 And here maven is resolving all versions of a version range: for ( Version version : versions ) https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/DefaultDependencyCollector.java#L405 Why not keep the same? We don't need to iterate over reversed list of versions, I think resolve the latest version by " Version version = versions.get( versions.size() - 1 ) " is enough. Any thoughts? With such change, no tests broken at all.
            ibabiankou Ivan Babiankou added a comment -

            wecai I'd say it makes sense. However it seems to me that it changes behaviour. I think that conflict resolution after loading all the versions might select different version, where as if it only loads the latest one the build might fail or just end up with different versions. I do want to draft couple of more complicated IT to better understand this, but didn't have time to do that yet.

            Anyway I wouldn't put this change together with the DFS > BFS. As michael-o pointed out, we should do one change at a time.

            ibabiankou Ivan Babiankou added a comment - wecai I'd say it makes sense. However it seems to me that it changes behaviour. I think that conflict resolution after loading all the versions might select different version, where as if it only loads the latest one the build might fail or just end up with different versions. I do want to draft couple of more complicated IT to better understand this, but didn't have time to do that yet. Anyway I wouldn't put this change together with the DFS > BFS. As michael-o pointed out, we should do one change at a time.

            Hello folks:

             

            Would it be feasible to hide the new behaviour behind a switch? If you are nervous about it.

             

            Additionally, I would be willing to try a preview build to verify that it resolves the issues I have reported.

             

            (I guess just pointing me to the PR branch would work too....)

             

            cannucklehead Gregory Ducharme added a comment - Hello folks:   Would it be feasible to hide the new behaviour behind a switch? If you are nervous about it.   Additionally, I would be willing to try a preview build to verify that it resolves the issues I have reported.   (I guess just pointing me to the PR branch would work too....)  
            michael-o Michael Osipov added a comment -

            ibabiankou, wecai, we have now merged a lot. Does it change the state of this ticket in any regard?

            michael-o Michael Osipov added a comment - ibabiankou , wecai , we have now merged a lot. Does it change the state of this ticket in any regard?
            jhughes Jim Hughes added a comment -

            michael-o I'm interested in this ticket as well.  When I looked at this related issue, I couldn't find an obvious way to write an integration test to show the issue with range resolution downloading all the dependencies in the range.

            If the recent work in the resolver hasn't addressed that issue, can you suggest some way to test the necessary changes?
            https://issues.apache.org/jira/browse/MNG-7049 

            jhughes Jim Hughes added a comment - michael-o I'm interested in this ticket as well.  When I looked at this related issue, I couldn't find an obvious way to write an integration test to show the issue with range resolution downloading all the dependencies in the range. If the recent work in the resolver hasn't addressed that issue, can you suggest some way to test the necessary changes? https://issues.apache.org/jira/browse/MNG-7049  

            Well, an ugly way to test this would be to override the maven logger and capture the output since maven does log everything it downloads.

            You would need to use an empty local repo.

            Then you could just add some logic to see if the optimal set of files was downloaded.

             

             

            cannucklehead Gregory Ducharme added a comment - Well, an ugly way to test this would be to override the maven logger and capture the output since maven does log everything it downloads. You would need to use an empty local repo. Then you could just add some logic to see if the optimal set of files was downloaded.    
            jhughes Jim Hughes added a comment -

            Is there an example of a test which already does that?  It sounds brittle as a test approach.

            jhughes Jim Hughes added a comment - Is there an example of a test which already does that?  It sounds brittle as a test approach.
            ibabiankou Ivan Babiankou added a comment -

            michael-o From the top of my head, I'd say the main points of the ticket are implemented: BFS, resolving newer versions first, and skipping resolution of the older versions.

            cannucklehead what do you think about this?

            jhughes Have a look at `MavenITmng3379ParallelArtifactDownloadsTest` it asserts artifacts are resolved (not sure if it's the best way to do that). IIUC you could do a similar thing: have a single dependency with a version range and expect only the expected artifact to be resolved.

            ibabiankou Ivan Babiankou added a comment - michael-o From the top of my head, I'd say the main points of the ticket are implemented: BFS, resolving newer versions first, and skipping resolution of the older versions. cannucklehead what do you think about this? jhughes Have a look at `MavenITmng3379ParallelArtifactDownloadsTest` it asserts artifacts are resolved (not sure if it's the best way to do that). IIUC you could do a similar thing: have a single dependency with a version range and expect only the expected artifact to be resolved.
            cannucklehead Gregory Ducharme added a comment - - edited

            Is there a way for me to obtain the proposed  resolver and test with my own use cases?

            cannucklehead Gregory Ducharme added a comment - - edited Is there a way for me to obtain the proposed  resolver and test with my own use cases?
            michael-o Michael Osipov added a comment -

            cannucklehead, yes. Install Resolver from master, package Maven 3.9.x from master with Resolver 1.8.0-SNAPSHOT and test.

            michael-o Michael Osipov added a comment - cannucklehead , yes. Install Resolver from master, package Maven 3.9.x from master with Resolver 1.8.0-SNAPSHOT and test.
            wecai wei cai added a comment - - edited

            michael-o cannucklehead ibabiankou 

            I think this ticket is now equivalent to https://issues.apache.org/jira/browse/MNG-7049, right? We just need to fix the unnecessary poms downloadings when comes to resolve a dependency with version range?

            The BFS  + Skipper is in BFDependencyCollector, as you can see with below code,  it will do skip resolution of lower versions, however it does not help skip pom/descriptor downloadings.

            https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java#L498

            To fix the unnecessary pom/descriptor downloadings, let's check the code:

            1. Maven downloads metadata.xml to figure out all available versions for the version range
            https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java#L384

            2. Download all poms/descriptors one by one
            https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java#L404

            The issue is in #2. Maven just downloads all poms matching thee version range, as Ivan pointed out in above threads, we can just reverse the version list and then break if any of the version (newer version comes first) is resolved successfully.

            Fired initial PR for discussion:
            https://github.com/apache/maven-resolver/pull/163

            Added you access, feel free to edit the PR if you would like to modify.

            Verified the patch by adding below dependency in my project and now only poms of gson-parent and gson 2.9.0 are downloaded.

            <dependency>
                <groupId>com.google.code.gson</groupId>
                <artifactId>gson</artifactId>
                <version>[0.0.1,3.0.0)</version>
            </dependency>
            wecai wei cai added a comment - - edited michael-o cannucklehead ibabiankou   I think this ticket is now equivalent to https://issues.apache.org/jira/browse/MNG-7049 , right? We just need to fix the unnecessary poms downloadings when comes to resolve a dependency with version range? The BFS  + Skipper is in BFDependencyCollector, as you can see with below code,  it will do skip resolution of lower versions, however it does not help skip pom/descriptor downloadings. https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java#L498 To fix the unnecessary pom/descriptor downloadings, let's check the code: 1. Maven downloads metadata.xml to figure out all available versions for the version range https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java#L384 2. Download all poms/descriptors one by one https://github.com/apache/maven-resolver/blob/master/maven-resolver-impl/src/main/java/org/eclipse/aether/internal/impl/collect/bf/BfDependencyCollector.java#L404 The issue is in #2. Maven just downloads all poms matching thee version range, as Ivan pointed out in above threads, we can just reverse the version list and then break if any of the version (newer version comes first) is resolved successfully. Fired initial PR for discussion: https://github.com/apache/maven-resolver/pull/163 Added you access, feel free to edit the PR if you would like to modify. Verified the patch by adding below dependency in my project and now only poms of gson-parent and gson 2.9.0 are downloaded. <dependency> <groupId>com.google.code.gson</groupId> <artifactId>gson</artifactId> <version>[0.0.1,3.0.0)</version> </dependency>
            michael-o Michael Osipov added a comment -

            The most important question: Is the versions array totally sorted? Consider you are maintaining 2.9.x and 3.0.x branches. The following releases have happened in chronological order: 2.9.1, 2.9.2, 3.0.0, 3.0.1, 2.9.3, 3.1.0, 3.0.2, 3.1.1, 2.9.4.

            michael-o Michael Osipov added a comment - The most important question: Is the versions array totally sorted? Consider you are maintaining 2.9.x and 3.0.x branches. The following releases have happened in chronological order: 2.9.1, 2.9.2, 3.0.0, 3.0.1, 2.9.3, 3.1.0, 3.0.2, 3.1.1, 2.9.4.
            wecai wei cai added a comment -

            michael-o 

            It already gets all versions from metadata.xml as defined in chronological order when creating the VersionRangeRequest, the most obsolete one comes the first.

            And the patch in above PR just reverse the list (Collections.reverse( versions )) so the latest deployed will come first (could be 2.9.x or 3.1.x depends on the deploy timestamp).

            wecai wei cai added a comment - michael-o   It already gets all versions from metadata.xml as defined in chronological order when creating the VersionRangeRequest, the most obsolete one comes the first. And the patch in above PR just reverse the list (Collections.reverse( versions )) so the latest deployed will come first (could be 2.9.x or 3.1.x depends on the deploy timestamp).

            Is this still open question? Resolver has DF (old) and BF (new) collectors implemented. Can this be closed out?

            cstamas Tamas Cservenak added a comment - Is this still open question? Resolver has DF (old) and BF (new) collectors implemented. Can this be closed out?
            wecai wei cai added a comment -

            Yes, I think it can be closed.

            发自我的iPhone

            wecai wei cai added a comment - Yes, I think it can be closed. 发自我的iPhone

            People

              michael-o Michael Osipov
              cannucklehead Gregory Ducharme
              Votes:
              1 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: