Maven
  1. Maven
  2. MNG-3472

configuration possibilities to limit size of local repository

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.0.8
    • Fix Version/s: None
    • Component/s: Settings
    • Labels:
      None

      Description

      it would be great to make repository-size configurable, for instance by setting the maximum number of downloads of a snapshot-version to be kept. thus the explosion of local repository size can be reduced.
      especially when you are working with many in-house multi-module projects which are marked as snapshots before released , can increase repository size significantly.

      see mailing-list discussion: http://www.nabble.com/limit-size-of-local-repository%2C-limit-number-of-snapshots-td16147475s177.html

        Issue Links

          Activity

          Hide
          Brett Porter added a comment -

          I think this is best handled by external tools.

          Show
          Brett Porter added a comment - I think this is best handled by external tools.
          Hide
          Georgios Skempes added a comment - - edited

          Unfortunately this issue was closed, otherwise I would have voted. I think every software should address the problems it creates. Pointing to external tools is not a solution. A software cannot assume that disk space is limitless available.

          Because of many configuration changes yesterday, my windows profile was grown up to a size of 16 GB. I didn't notice. Since the profile is stored on the server, it will take more than 4.5 hours to login today (the network works only at 10 Mbit).

          Please reopen this issue and provide a solution. I'm sure you would do a favor to many people.

          Show
          Georgios Skempes added a comment - - edited Unfortunately this issue was closed, otherwise I would have voted. I think every software should address the problems it creates. Pointing to external tools is not a solution. A software cannot assume that disk space is limitless available. Because of many configuration changes yesterday, my windows profile was grown up to a size of 16 GB. I didn't notice. Since the profile is stored on the server, it will take more than 4.5 hours to login today (the network works only at 10 Mbit). Please reopen this issue and provide a solution. I'm sure you would do a favor to many people.
          Hide
          Joerg Schaible added a comment -

          Maybe you should simply start reading docs for the Maven settings. I am sure that quite a lot of people do not keep the local repo in their profile ...

          Show
          Joerg Schaible added a comment - Maybe you should simply start reading docs for the Maven settings. I am sure that quite a lot of people do not keep the local repo in their profile ...
          Hide
          David Greenberg added a comment -

          Joerg, would you please expand upon what you mean by not keeping the local repo in the profile? While I'm sure the "documentation" has plenty of information, it is always helpful to summarize or at least link to the specific text that is relevant to the problem.

          I also would have voted for this, as 14GB mysteriously disappeared from my server over time until I thought to look at the local repo. An external tool would be fine; does one exist that fixes precisely this problem?

          Show
          David Greenberg added a comment - Joerg, would you please expand upon what you mean by not keeping the local repo in the profile? While I'm sure the "documentation" has plenty of information, it is always helpful to summarize or at least link to the specific text that is relevant to the problem. I also would have voted for this, as 14GB mysteriously disappeared from my server over time until I thought to look at the local repo. An external tool would be fine; does one exist that fixes precisely this problem?
          Hide
          Benjamin Bentmann added a comment -

          it is always helpful to summarize or at least link to the specific text that is relevant to the problem.

          The Settings Reference describes the <localRepository> element which can be used to specify a custom path for the local repo.

          Show
          Benjamin Bentmann added a comment - it is always helpful to summarize or at least link to the specific text that is relevant to the problem. The Settings Reference describes the <localRepository> element which can be used to specify a custom path for the local repo.
          Hide
          Andrus Adamchik added a comment -

          I take issue with Brett's initial comment: "I think this is best handled by external tools."... On a local machine Maven is the tool to manage repository. At least the way things stand now... I guess since the issue got closed, the easiest solution in the spirit of open source would be to write some kind of a "maven-vacuum-repo-plugin" (something I may end up doing, but no time guarantees)...

          Show
          Andrus Adamchik added a comment - I take issue with Brett's initial comment: "I think this is best handled by external tools."... On a local machine Maven is the tool to manage repository. At least the way things stand now... I guess since the issue got closed, the easiest solution in the spirit of open source would be to write some kind of a "maven-vacuum-repo-plugin" (something I may end up doing, but no time guarantees)...
          Hide
          David Greenberg added a comment -

          Andrus is right. The local repository is fundamentally tied to the Maven tool; therefore, Maven should manage it. A plugin would do the job, but I think there is more to this than simply scrubbing the repository periodically. I would be concerned that cleaning the local repo would eventually slow down builds. After all, why have a local repo if it doesn't make the build process more efficient? What would be really great is a way for Maven to determine which dependencies are used most often. Then, it could keep those while cleaning up the rest. Whatever extension to Maven that does this should clean the repo before downloading any new files. That way, when a critical bug fix needs to be built and tested, the developer or build server is not waiting forever for the local repo sweep to finish. It may make each build just slightly longer, but it would prevent bottlenecks.

          Just my two cents...and I understand that I should just build it instead of complaining about it. I'll see what I can do.

          Show
          David Greenberg added a comment - Andrus is right. The local repository is fundamentally tied to the Maven tool; therefore, Maven should manage it. A plugin would do the job, but I think there is more to this than simply scrubbing the repository periodically. I would be concerned that cleaning the local repo would eventually slow down builds. After all, why have a local repo if it doesn't make the build process more efficient? What would be really great is a way for Maven to determine which dependencies are used most often. Then, it could keep those while cleaning up the rest. Whatever extension to Maven that does this should clean the repo before downloading any new files. That way, when a critical bug fix needs to be built and tested, the developer or build server is not waiting forever for the local repo sweep to finish. It may make each build just slightly longer, but it would prevent bottlenecks. Just my two cents...and I understand that I should just build it instead of complaining about it. I'll see what I can do.
          Hide
          Kariem Hussein added a comment -

          I have to agree with Andrus and David. The question is not how to configure maven to point to a local repository, but to configure maven to manage the space used by the local repository. The repository is maven's responsibility and should not be handled by external tools.

          As can be seen from the linked issue CONTINUUM-1693 (referring to CONTINUUM-782), projects relying on maven have to build workarounds to handle this.

          What can be done to raise awareness and reopen this ticket? What is needed to include something like this?

          Show
          Kariem Hussein added a comment - I have to agree with Andrus and David. The question is not how to configure maven to point to a local repository, but to configure maven to manage the space used by the local repository. The repository is maven's responsibility and should not be handled by external tools. As can be seen from the linked issue CONTINUUM-1693 (referring to CONTINUUM-782 ), projects relying on maven have to build workarounds to handle this. What can be done to raise awareness and reopen this ticket? What is needed to include something like this?
          Hide
          Matthew Corby-Eaglen added a comment -

          I too think this should be maven's responsibility. Why has this bug / feature request been dismissed out of hand?

          If you had a disk logger or any piece of software creating a tonne of garbage on your system, you would expect to have a setting or switch which would allow you to control the amount it creates.

          As it is, we have to create scripts to delete the repo, or parts of it, every so often, which is a royal pain. To me, it does not matter where the repo sits, I just cannot have it growing unchecked.

          Please reopen this issue, and listen to the community.

          Show
          Matthew Corby-Eaglen added a comment - I too think this should be maven's responsibility. Why has this bug / feature request been dismissed out of hand? If you had a disk logger or any piece of software creating a tonne of garbage on your system, you would expect to have a setting or switch which would allow you to control the amount it creates. As it is, we have to create scripts to delete the repo, or parts of it, every so often, which is a royal pain. To me, it does not matter where the repo sits, I just cannot have it growing unchecked. Please reopen this issue, and listen to the community.
          Hide
          Paul Benedict added a comment -

          I do not think this is Maven's responsibility in the slightest. Rather, it should be the responsibility of CI tools like Hudson or Continuum to clear out older contents; I would make an RFE over at those projects. As a last resort, you have help from your operating system: disk quotas and custom batch processes.

          Show
          Paul Benedict added a comment - I do not think this is Maven's responsibility in the slightest. Rather, it should be the responsibility of CI tools like Hudson or Continuum to clear out older contents; I would make an RFE over at those projects. As a last resort, you have help from your operating system: disk quotas and custom batch processes.
          Hide
          François Duchatelet added a comment -

          There are two sides on this issue:

          • but the triggering of the action is the responsibility of an external tool, such as hudson, continuum, cron, ...

          François

          Show
          François Duchatelet added a comment - There are two sides on this issue: as the local repository is managed by maven, maven should provide some tools to control its size, content, ... Writing any kind of script to do some clever cleanup requires knowledge of repo structure. Partial answer, the dependency:purge-local-repository plugin ( http://maven.apache.org/plugins/maven-dependency-plugin/purge-local-repository-mojo.html ). Maybe this one should be extended with other kind of cleanup ? but the triggering of the action is the responsibility of an external tool, such as hudson, continuum, cron, ... François
          Hide
          Wendy Smoak added a comment -

          This issue is closed, so nothing is likely to change unless someone brings it up on the mailing list for discussion.

          Continuum already has a local repository purge feature. Perhaps that code could be turned into a separate utility/daemon.

          Show
          Wendy Smoak added a comment - This issue is closed, so nothing is likely to change unless someone brings it up on the mailing list for discussion. Continuum already has a local repository purge feature. Perhaps that code could be turned into a separate utility/daemon.
          Hide
          Brian Levine added a comment -

          Wow. I'm really surprised at the responses here. While it is true that CI systems could provide tools to clean out Maven local repos, this doesn't do much for the individual developer who finds that all of a sudden, his dev machine has run out of disk space. This seems like a perfect example of something that should be configurable in settings.xml with a default value that is fairly small. On a personal development machine especially, I see no reason why you'd want to keep old snapshot artifacts. I will post something to the mailing list to get this issue re-opened.

          Show
          Brian Levine added a comment - Wow. I'm really surprised at the responses here. While it is true that CI systems could provide tools to clean out Maven local repos, this doesn't do much for the individual developer who finds that all of a sudden, his dev machine has run out of disk space. This seems like a perfect example of something that should be configurable in settings.xml with a default value that is fairly small. On a personal development machine especially, I see no reason why you'd want to keep old snapshot artifacts. I will post something to the mailing list to get this issue re-opened.
          Hide
          dhesson added a comment -

          I also think this should be a configured responsibility of Maven. Maven is the only client that knows how the repository should look, especially if it ever evolves and changes the structure. Leaving it up to external tools could corrupt repositories. I know right now the repository is straight forward, but still. The tool that generated it should know how to handle it.

          Show
          dhesson added a comment - I also think this should be a configured responsibility of Maven. Maven is the only client that knows how the repository should look, especially if it ever evolves and changes the structure. Leaving it up to external tools could corrupt repositories. I know right now the repository is straight forward, but still. The tool that generated it should know how to handle it.
          Hide
          Eric Dalquist added a comment -

          It is unfortunate that this issue is marked as won't-fix. We run into this problem all the time and are starting to feel lots of pain around managing local repositories. Having some way to:

          • Purge all but the X newest snapshots of a version
          • Purge snapshots where a release exists in the local repo

          Is really critical, for now we'll be doing this via scripts which I'll try to post back here but of course the danger is the local repository meta data may be left in an invalid state since we'll just be deleting files.

          Show
          Eric Dalquist added a comment - It is unfortunate that this issue is marked as won't-fix. We run into this problem all the time and are starting to feel lots of pain around managing local repositories. Having some way to: Purge all but the X newest snapshots of a version Purge snapshots where a release exists in the local repo Is really critical, for now we'll be doing this via scripts which I'll try to post back here but of course the danger is the local repository meta data may be left in an invalid state since we'll just be deleting files.
          Hide
          Terence Mill added a comment - - edited

          I think maven should provide at least the method to be able t purge the local repository similar to the nexus server option (number of snapshots, release and days to keep, delete spnaphosts if released), also this job must be triggered from external tool, like ej. jenkins or devscript.
          Would be great if someone can posts plugin or scripts which can do such work in the meantime.

          Here is my answer:
          http://stackoverflow.com/questions/1308263/wiping-out-maven-local-repository-on-build-machine

          Pleas repopen this issue.

          Show
          Terence Mill added a comment - - edited I think maven should provide at least the method to be able t purge the local repository similar to the nexus server option (number of snapshots, release and days to keep, delete spnaphosts if released), also this job must be triggered from external tool, like ej. jenkins or devscript. Would be great if someone can posts plugin or scripts which can do such work in the meantime. Here is my answer: http://stackoverflow.com/questions/1308263/wiping-out-maven-local-repository-on-build-machine Pleas repopen this issue.
          Hide
          Eric Dalquist added a comment - - edited

          Here is what we are doing on our build server:

          find /path/to/.m2 -wholename '*SNAPSHOT/*.*' -type f -mtime +1 -print0 | xargs -r -0 rm -v
          find /path/to/.m2 -type d -empty -print0 | xargs -r -0 rmdir -v
          

          That deletes any files in a SNAPSHOT directory older than 1 day and then deletes all empty directories.

          Show
          Eric Dalquist added a comment - - edited Here is what we are doing on our build server: find /path/to/.m2 -wholename '*SNAPSHOT/*.*' -type f -mtime +1 -print0 | xargs -r -0 rm -v find /path/to/.m2 -type d -empty -print0 | xargs -r -0 rmdir -v That deletes any files in a SNAPSHOT directory older than 1 day and then deletes all empty directories.

            People

            • Assignee:
              Unassigned
              Reporter:
              manuel aldana
            • Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development