Maven
  1. Maven
  2. MNG-4142

Maven doesn't try to download a dependency when it already exists a version with classifier locally

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 2.0.9, 2.0.10, 2.1.0
    • Component/s: Dependencies
    • Labels:
      None
    • Environment:
      Java 5, Windows XP

      Description

      When maven installs in the local repository an artifact with a classifier, and not the principal artifact, it won't try in a reactor to download the principal artifact from the remote repository.
      I created a testcase to reproduce it. It's quite simple. You unzip it and in the root directory you launch

      mvn site

      This build uses clover, thus it installs locally cloverified versions of artifacts.
      The build will fail because it doesn't find the SNAPSHOT of the module1 :

      [INFO] ------------------------------------------------------------------------
      [ERROR] BUILD ERROR
      [INFO] ------------------------------------------------------------------------
      [INFO] Failed to resolve artifact.
      
      Missing:
      ----------
      1) org.apache.maven.it.test-metadata-local-clover:module1:jar:1.0.0-SNAPSHOT
      
        Try downloading the file manually from the project website.
      
        Then, install it using the command:
            mvn install:install-file -DgroupId=org.apache.maven.it.test-metadata-local-clover -DartifactId=module1 -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar -Dfile=/pa
      th/to/file
      
        Alternatively, if you host your own repository you can deploy the file there:
            mvn deploy:deploy-file -DgroupId=org.apache.maven.it.test-metadata-local-clover -DartifactId=module1 -Dversion=1.0.0-SNAPSHOT -Dpackaging=jar -Dfile=/path
      /to/file -Durl=[url] -DrepositoryId=[id]
      
        Path to dependency:
              1) org.apache.maven.it.test-metadata-local-clover:module2:jar:1.0.0-SNAPSHOT
              2) org.apache.maven.it.test-metadata-local-clover:module1:jar:1.0.0-SNAPSHOT
      
      ----------
      1 required artifact is missing.
      
      for artifact:
        org.apache.maven.it.test-metadata-local-clover:module2:jar:1.0.0-SNAPSHOT
      
      from the specified remote repositories:
        central (http://maven-proxy.groupe.generali.fr/all),
        remote-repo (file://E:\jtb\workspaces\tests\test-metadata-local-clover/remote-repo)
      

      It's anormal because I set a "local" remote repository in the build where it should try to download it.
      You can validate that it is working by launching :

      mvn install -f module2/pom.xml

      This bug is really annoying for us. It exists for a long long time now. I thought it was due to a problem with metadata sent by archiva, but after migrating to nexus the problem always occurs.
      I don't know if the problem is in metadata or in the reactor itself. I think it is the second one. There are many issues opened about the reactor and classifiers.
      Is there some who can try if it can reproduce the error with my testcase ?
      It should be easy to create a real IT for maven with it if it is necessary.

      1. test-metadata-local-clover.zip
        21 kB
        Arnaud HERITIER
      2. MNG-4142.patch
        2 kB
        David Rousselie

        Activity

        Arnaud HERITIER created issue -
        Hide
        Arnaud HERITIER added a comment -

        Something else to add : If I launch after the site build, another build which doesn't produce the main artefact (for example : mvn sonar), it will also fail trying to download dependency. I think that it can be related to maven-metadata and more probably to the maven maven-metadata-local.xml which is generated.
        After the mvn site, I have in my local repository in org\apache\maven\it\test-metadata-local-clover\module1\1.0.0-SNAPSHOT:

        maven-metadata-central.xml
        maven-metadata-local.xml
        maven-metadata-remote-repo.xml
        maven-metadata-remote-repo.xml.sha1
        module1-1.0.0-SNAPSHOT-clover.jar
        module1-1.0.0-SNAPSHOT.pom
        

        and in each metadata file :

        • maven-metadata-central.xml
          <?xml version="1.0" encoding="UTF-8"?>
          <metadata xsi:schemaLocation="http://maven.apache.org/METADATA/1.0.0 http://maven.apache.org/xsd/metadata-1.0.0.xsd" xmlns="http://maven.apache.org/METADATA/1.0.0"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <groupId>org.apache.maven.it.test-metadata-local-clover</groupId>
            <artifactId>module1</artifactId>
            <version>1.0.0-SNAPSHOT</version>
          </metadata>
          
        • maven-metadata-local.xml
          <?xml version="1.0" encoding="UTF-8"?>
          <metadata xsi:schemaLocation="http://maven.apache.org/METADATA/1.0.0 http://maven.apache.org/xsd/metadata-1.0.0.xsd" xmlns="http://maven.apache.org/METADATA/1.0.0"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <groupId>org.apache.maven.it.test-metadata-local-clover</groupId>
            <artifactId>module1</artifactId>
            <version>1.0.0-SNAPSHOT</version>
            <versioning>
              <snapshot>
                <localCopy>true</localCopy>
              </snapshot>
              <lastUpdated>20090423144418</lastUpdated>
            </versioning>
          </metadata>
          
        • maven-metadata-remote-repo.xml
          <?xml version="1.0" encoding="UTF-8"?>
          <metadata xsi:schemaLocation="http://maven.apache.org/METADATA/1.0.0 http://maven.apache.org/xsd/metadata-1.0.0.xsd" xmlns="http://maven.apache.org/METADATA/1.0.0"
              xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <groupId>org.apache.maven.it.test-metadata-local-clover</groupId>
            <artifactId>module1</artifactId>
            <version>1.0.0-SNAPSHOT</version>
            <versioning>
              <snapshot>
                <timestamp>20090423.140624</timestamp>
                <buildNumber>1</buildNumber>
              </snapshot>
              <lastUpdated>20090423140625</lastUpdated>
            </versioning>
          </metadata>
          

          Do you think that maven-metadata-local.xml is good ? It says that there's a local copy whereas there's no module1-1.0.0-SNAPSHOT.jar in the directory.

        Show
        Arnaud HERITIER added a comment - Something else to add : If I launch after the site build, another build which doesn't produce the main artefact (for example : mvn sonar), it will also fail trying to download dependency. I think that it can be related to maven-metadata and more probably to the maven maven-metadata-local.xml which is generated. After the mvn site, I have in my local repository in org\apache\maven\it\test-metadata-local-clover\module1\1.0.0-SNAPSHOT: maven-metadata-central.xml maven-metadata-local.xml maven-metadata-remote-repo.xml maven-metadata-remote-repo.xml.sha1 module1-1.0.0-SNAPSHOT-clover.jar module1-1.0.0-SNAPSHOT.pom and in each metadata file : maven-metadata-central.xml <?xml version= "1.0" encoding= "UTF-8" ?> <metadata xsi:schemaLocation= "http: //maven.apache.org/METADATA/1.0.0 http://maven.apache.org/xsd/metadata-1.0.0.xsd" xmlns= "http://maven.apache.org/METADATA/1.0.0" xmlns:xsi= "http: //www.w3.org/2001/XMLSchema-instance" > <groupId>org.apache.maven.it.test-metadata-local-clover</groupId> <artifactId>module1</artifactId> <version>1.0.0-SNAPSHOT</version> </metadata> maven-metadata-local.xml <?xml version= "1.0" encoding= "UTF-8" ?> <metadata xsi:schemaLocation= "http: //maven.apache.org/METADATA/1.0.0 http://maven.apache.org/xsd/metadata-1.0.0.xsd" xmlns= "http://maven.apache.org/METADATA/1.0.0" xmlns:xsi= "http: //www.w3.org/2001/XMLSchema-instance" > <groupId>org.apache.maven.it.test-metadata-local-clover</groupId> <artifactId>module1</artifactId> <version>1.0.0-SNAPSHOT</version> <versioning> <snapshot> <localCopy> true </localCopy> </snapshot> <lastUpdated>20090423144418</lastUpdated> </versioning> </metadata> maven-metadata-remote-repo.xml <?xml version= "1.0" encoding= "UTF-8" ?> <metadata xsi:schemaLocation= "http: //maven.apache.org/METADATA/1.0.0 http://maven.apache.org/xsd/metadata-1.0.0.xsd" xmlns= "http://maven.apache.org/METADATA/1.0.0" xmlns:xsi= "http: //www.w3.org/2001/XMLSchema-instance" > <groupId>org.apache.maven.it.test-metadata-local-clover</groupId> <artifactId>module1</artifactId> <version>1.0.0-SNAPSHOT</version> <versioning> <snapshot> <timestamp>20090423.140624</timestamp> <buildNumber>1</buildNumber> </snapshot> <lastUpdated>20090423140625</lastUpdated> </versioning> </metadata> Do you think that maven-metadata-local.xml is good ? It says that there's a local copy whereas there's no module1-1.0.0-SNAPSHOT.jar in the directory.
        John Casey made changes -
        Field Original Value New Value
        Fix Version/s 2.2.x [ 13141 ]
        Hide
        David Rousselie added a comment -

        The problem is that when the Maven resolve the artifact version with maven-metadata-local.xml when it exists and it resolves it to 1.0.0-SNAPSHOT whereas only module1-1.0.0-SNAPSHOT-clover.jar (in this exemple). Then when it try to access the file artifact (module1-1.0.0-SNAPSHOT.jar) and fails, it tries to get the same version on remote repositories whereas it should have try to resolve the artifact version with data found in maven-metadata-$

        {remotereponame}

        .xml before trying to get it from each repository.

        I have attached a small patch (base on revision 805295 in branch maven-2.2.x) which might not be the best way to correct this bug : while resolving artifact version, it loads local metadata only if artifact's file exists locally.

        Show
        David Rousselie added a comment - The problem is that when the Maven resolve the artifact version with maven-metadata-local.xml when it exists and it resolves it to 1.0.0-SNAPSHOT whereas only module1-1.0.0-SNAPSHOT-clover.jar (in this exemple). Then when it try to access the file artifact (module1-1.0.0-SNAPSHOT.jar) and fails, it tries to get the same version on remote repositories whereas it should have try to resolve the artifact version with data found in maven-metadata-$ {remotereponame} .xml before trying to get it from each repository. I have attached a small patch (base on revision 805295 in branch maven-2.2.x) which might not be the best way to correct this bug : while resolving artifact version, it loads local metadata only if artifact's file exists locally.
        Hide
        David Rousselie added a comment -

        A possible correction for bug MNG-4142

        Show
        David Rousselie added a comment - A possible correction for bug MNG-4142
        David Rousselie made changes -
        Attachment MNG-4142.patch [ 43860 ]
        Hide
        Peter Liljenberg added a comment -

        I've encountered this problem as well when doing distributed builds (using Hudson).
        When one build server (#1) builds and deploys an artifact the maven-metadata-local.xml will contain the localCopy attribute:
        <versioning>
        <snapshot>
        <localCopy>true</localCopy>
        </snapshot>
        <lastUpdated>20091127124227</lastUpdated>
        </versioning>

        Which means that if another build server builds the same module AFTER server #1 and produce a new artifact in the repository, machine #1 will never download this new SNAPSHOT version when trying to build artifacts depending on the artifact.
        This is a huge problem (and a scary one!) for us and I would like to see a fix for this soon

        Show
        Peter Liljenberg added a comment - I've encountered this problem as well when doing distributed builds (using Hudson). When one build server (#1) builds and deploys an artifact the maven-metadata-local.xml will contain the localCopy attribute: <versioning> <snapshot> <localCopy>true</localCopy> </snapshot> <lastUpdated>20091127124227</lastUpdated> </versioning> Which means that if another build server builds the same module AFTER server #1 and produce a new artifact in the repository, machine #1 will never download this new SNAPSHOT version when trying to build artifacts depending on the artifact. This is a huge problem (and a scary one!) for us and I would like to see a fix for this soon
        Hide
        Brian Laframboise added a comment - - edited

        I just ran into this issue myself (with Maven 2.2.1). A build on one of my build machines was failing due to maven not downloading a newer version of a dependency that was available on Nexus because it had previously installed an earlier version of that same dependency into the local repository.

        When I removed the <localCopy>true</localCopy> element from the metadata of the non-updating dependency the problem went away - and dutifully came back when I re-added it.

        Peter, I agree - this is a scary problem for anybody trying to implement a distributed build system. What I don't understand is why more people are not encountering this issue.

        For now I'll attempt a workaround this by adding a cron job, per build agent, to frequently change <localCopy> back to false. If anybody has a better suggestion, I'd love to hear it.

        Show
        Brian Laframboise added a comment - - edited I just ran into this issue myself (with Maven 2.2.1). A build on one of my build machines was failing due to maven not downloading a newer version of a dependency that was available on Nexus because it had previously installed an earlier version of that same dependency into the local repository. When I removed the <localCopy>true</localCopy> element from the metadata of the non-updating dependency the problem went away - and dutifully came back when I re-added it. Peter, I agree - this is a scary problem for anybody trying to implement a distributed build system. What I don't understand is why more people are not encountering this issue. For now I'll attempt a workaround this by adding a cron job, per build agent, to frequently change <localCopy> back to false. If anybody has a better suggestion, I'd love to hear it.
        Brett Porter made changes -
        Fix Version/s 2.2.x (to be reviewed) [ 13141 ]
        Fix Version/s Issues to be reviewed for 3.x [ 13145 ]
        Hide
        Martin Todorov added a comment -

        Hi,

        I would like to ask why the attached patch is not being accepted?
        There are currently over three dozen votes for this and it is apparent that people are hitting the problem.

        In addition, perhaps the title of the bug is not quite correct. This doesn't just happen for artifacts with classifiers. In our case it happens for POM files where there are several inherited POMs.
        We're using Hudson and Nexus had to limit the parents to build on one Hudson node, while all modules which depend on them - on the rest of the nodes.

        I also think this is a huge and very obscure problem (esp. with distributed builds) and, provided that there is already a patch, I believe it's about time for it to be reviewed.

        Regards,

        Martin

        Show
        Martin Todorov added a comment - Hi, I would like to ask why the attached patch is not being accepted? There are currently over three dozen votes for this and it is apparent that people are hitting the problem. In addition, perhaps the title of the bug is not quite correct. This doesn't just happen for artifacts with classifiers. In our case it happens for POM files where there are several inherited POMs. We're using Hudson and Nexus had to limit the parents to build on one Hudson node, while all modules which depend on them - on the rest of the nodes. I also think this is a huge and very obscure problem (esp. with distributed builds) and, provided that there is already a patch, I believe it's about time for it to be reviewed. Regards, Martin
        Hide
        Arnaud HERITIER added a comment -

        Were you able to test if the problem exists with maven 3 ?
        I agree that this issue is really annoying but for now there is no 2.X version release scheduled.
        That's why there was no feedback from dev team.

        Show
        Arnaud HERITIER added a comment - Were you able to test if the problem exists with maven 3 ? I agree that this issue is really annoying but for now there is no 2.X version release scheduled. That's why there was no feedback from dev team.
        Hide
        Martin Todorov added a comment -

        Bonjour Arnaud!

        As per your request, I have tested this with Maven 3.0 today. I am afraid the problem still persists. Today this actually happened with a completely different project/setup/machine under both Maven 2.2.1 and 3.0.

        This is not just annoying, it's actually a show-stopper when it happens to a developer who's not expecting it, as they spend quite some time trying to figure it out.

        Please review for the release of Maven 3.1, (if a 2.x version will not be further released).

        Regards,

        Martin

        Show
        Martin Todorov added a comment - Bonjour Arnaud! As per your request, I have tested this with Maven 3.0 today. I am afraid the problem still persists. Today this actually happened with a completely different project/setup/machine under both Maven 2.2.1 and 3.0. This is not just annoying, it's actually a show-stopper when it happens to a developer who's not expecting it, as they spend quite some time trying to figure it out. Please review for the release of Maven 3.1, (if a 2.x version will not be further released). Regards, Martin
        Hide
        nodje added a comment -

        I've the same problem but with standard SNAPSHOT dependencies, not using classifier, and with 3.0.3.

        Show
        nodje added a comment - I've the same problem but with standard SNAPSHOT dependencies, not using classifier, and with 3.0.3.
        Hide
        Martin Todorov added a comment - - edited

        This bug/feature doesn't really have much to do with classifiers. I believe the original bug report filer just thought it had something to do with the problem.
        Perhaps somebody should rename the bug to "Maven doesn't try to download SNAPSHOT artifacts, if you've installed them locally; even if you pass -U".

        I believe that by using -U the local metadata should get "unlocked" (which currently doesn't do this) for the locally installed SNAPSHOT artifacts.

        Show
        Martin Todorov added a comment - - edited This bug/feature doesn't really have much to do with classifiers. I believe the original bug report filer just thought it had something to do with the problem. Perhaps somebody should rename the bug to "Maven doesn't try to download SNAPSHOT artifacts, if you've installed them locally; even if you pass -U". I believe that by using -U the local metadata should get "unlocked" (which currently doesn't do this) for the locally installed SNAPSHOT artifacts.
        Hide
        nodje added a comment -

        alright, good then, that's my problem!

        Can't wait for the fix to come, hopefully as early as 3.0.4...

        Show
        nodje added a comment - alright, good then, that's my problem! Can't wait for the fix to come, hopefully as early as 3.0.4...
        Hide
        nodje added a comment - - edited

        We're a year after the latest comment, could someone update the Fix version, that'd give some hope.

        Our team is severely being affected by this problem, and it seems to be larger than the description suggests.
        We have SNAPSHOT update problems on a daily basis, even for deps than haven't been installed locally.
        A CI server deploy the artifact, and in some case no one seem to get the updated artifact.

        Thanks for pointing me to other bug report if my problem is out of scope here.

        Show
        nodje added a comment - - edited We're a year after the latest comment, could someone update the Fix version, that'd give some hope. Our team is severely being affected by this problem, and it seems to be larger than the description suggests. We have SNAPSHOT update problems on a daily basis, even for deps than haven't been installed locally. A CI server deploy the artifact, and in some case no one seem to get the updated artifact. Thanks for pointing me to other bug report if my problem is out of scope here.
        Hide
        Martin Todorov added a comment -

        I have replied to numerous posts on Stack Overflow concerning this particular issue. People keep hitting this and are unhappy with the current behavior. In the company I work for, we had to make Jenkins use private repositories for each project which would be removed before every build, just for the sake of sanity and for us not losing our minds when a build fails due to this reason.

        The problem here is also observed under Maven 3. Furthermore, it's not a classifier problem. It happens to whatever dependencies you have to modules which you've installed locally.

        My suggestion would be to make the -U option "unlock" the Maven metadata files so that they are fetched from the remote repository.

        I would also appreciate some response from the you guys. What's the problem here that keeps this issue unscheduled for future releases? We haven't received an official comment for a long while. :-|

        Show
        Martin Todorov added a comment - I have replied to numerous posts on Stack Overflow concerning this particular issue. People keep hitting this and are unhappy with the current behavior. In the company I work for, we had to make Jenkins use private repositories for each project which would be removed before every build, just for the sake of sanity and for us not losing our minds when a build fails due to this reason. The problem here is also observed under Maven 3. Furthermore, it's not a classifier problem. It happens to whatever dependencies you have to modules which you've installed locally. My suggestion would be to make the -U option "unlock" the Maven metadata files so that they are fetched from the remote repository. I would also appreciate some response from the you guys. What's the problem here that keeps this issue unscheduled for future releases? We haven't received an official comment for a long while. :-|
        Hide
        nodje added a comment -

        Removing a private repository before each build would indeed solve my problem
        I'm considering the option, since I'm losing my mind with all the build failing due to SNAPSHOT update problem.

        Developers new to Maven in our team don't even understand "SNAPSHOT update", they just think they have to install manually each SNAPSHOT dependency... which messes up even more the whole thing.

        And I insist, our CI server build agents don't install SNAPSHOT locally, and still they fail to retrieve the latest snapshots in many cases.

        It's not a locally installed artifact update problem ONLY.

        Show
        nodje added a comment - Removing a private repository before each build would indeed solve my problem I'm considering the option, since I'm losing my mind with all the build failing due to SNAPSHOT update problem. Developers new to Maven in our team don't even understand "SNAPSHOT update", they just think they have to install manually each SNAPSHOT dependency... which messes up even more the whole thing. And I insist, our CI server build agents don't install SNAPSHOT locally, and still they fail to retrieve the latest snapshots in many cases. It's not a locally installed artifact update problem ONLY.
        Hide
        Cody Fyler added a comment -

        We are experiencing this issue with Maven 3.0.5 on Jenkins with multiple slaves.

        It's kind of a nightmare.

        Can we please get this at least assigned?

        Here is the workaround that I came up with that in my opinion really sucks, but it works.

        http://maven.apache.org/plugins/maven-dependency-plugin/purge-local-repository-mojo.html#include

        In this case, I use a filter to clear out the SNAPSHOT artifacts that need to be updated. But it sucks that I have to do it. And we have a LOT of projects where we are going to have to do something similar.

        Show
        Cody Fyler added a comment - We are experiencing this issue with Maven 3.0.5 on Jenkins with multiple slaves. It's kind of a nightmare. Can we please get this at least assigned? Here is the workaround that I came up with that in my opinion really sucks, but it works. http://maven.apache.org/plugins/maven-dependency-plugin/purge-local-repository-mojo.html#include In this case, I use a filter to clear out the SNAPSHOT artifacts that need to be updated. But it sucks that I have to do it. And we have a LOT of projects where we are going to have to do something similar.
        Hide
        nodje added a comment -

        Good luck, you just have to live with it or spend the time to understand the Maven sources by yourself.
        See Martin Todorov comments for a work around.

        Show
        nodje added a comment - Good luck, you just have to live with it or spend the time to understand the Maven sources by yourself. See Martin Todorov comments for a work around.
        Hide
        John Chatham added a comment -

        Ran into this issue recently on maven 3.0.4; our workaround was to update to maven 3.2.1, where there's a new property one can set: maven.install.skip

        Set that to true, either with -Dmaven.install.skip or (how we did it) by adding a property entry in the maven settings file our jenkins build was using, and that fixed that; we build and deploy our snapshots, but don't install them into the local repositories.

        Show
        John Chatham added a comment - Ran into this issue recently on maven 3.0.4; our workaround was to update to maven 3.2.1, where there's a new property one can set: maven.install.skip Set that to true, either with -Dmaven.install.skip or (how we did it) by adding a property entry in the maven settings file our jenkins build was using, and that fixed that; we build and deploy our snapshots, but don't install them into the local repositories.
        Mark Thomas made changes -
        Project Import Sun Apr 05 08:49:45 UTC 2015 [ 1428223785911 ]
        Mark Thomas made changes -
        Workflow jira [ 12714490 ] Default workflow, editable Closed status [ 12754197 ]
        Mark Thomas made changes -
        Project Import Sun Apr 05 21:45:26 UTC 2015 [ 1428270326204 ]
        Mark Thomas made changes -
        Workflow jira [ 12952529 ] Default workflow, editable Closed status [ 12989870 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Arnaud HERITIER
          • Votes:
            36 Vote for this issue
            Watchers:
            19 Start watching this issue

            Dates

            • Created:
              Updated:

              Development