Continuum
  1. Continuum
  2. CONTINUUM-1693

Continuum fills our server disk with SNAPSHOTs.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2
    • Component/s: None
    • Labels:
      None

      Description

      Our Deployment Repository Directory fills our server disk with SNAPSHOTs after running hourly builds for a couple of weeks.

      We have more or less a hundred applications configured in Continuum. A lot of them create assemblies with all their dependencies embedded, so each snapshot deployed eats a couple of megabytes. We also need hourly builds.

      We use 'maven clean deploy' to deploy these snapshots to a dedicated Archiva snapshot repository. In Archiva, these snapshots get purged after a week, only keeping the last 2 or 3 snapshot versions.

      However, continuum always installs to the local repository first, before uploading them to Archiva. This repository never gets purged. I was thinking to have Continuum use Archiva's repository directly as internal repository, but I don't know if that's safe or not. Archiva won't know these changes before it scans the repository again.

      I was also looking at maven-dependency-plugin:purge-local-repository, but that's not exactly what we want. We just want to purge old snapshot versions when new snapshot versions are installed. Maybe that could be a general Maven feature, but it is especially important for Continuum.

        Issue Links

          Activity

          Hide
          Geert Pante added a comment -

          This never made it into a release...

          Show
          Geert Pante added a comment - This never made it into a release...
          Hide
          Wendy Smoak added a comment -

          What version of Continuum are you using? The 'Affects Version' was not specified.

          The summary mentions the deployment repository directory, but the description talks about the local repository. Which one are you concerned about?

          The 'Deployment Repository Directory' (third item on the configuration page) is optional. If you have a separate remote repository, you probably want to leave that field blank.

          The local repository is not optional, though you could clear it out on whatever schedule suits you. This is the same as a developer occasionally deleting his local repository.

          A tool to purge the local repository the same way Archiva does for its managed repos would be nice, but is not specific to Continuum.

          Show
          Wendy Smoak added a comment - What version of Continuum are you using? The 'Affects Version' was not specified. The summary mentions the deployment repository directory, but the description talks about the local repository. Which one are you concerned about? The 'Deployment Repository Directory' (third item on the configuration page) is optional. If you have a separate remote repository, you probably want to leave that field blank. The local repository is not optional, though you could clear it out on whatever schedule suits you. This is the same as a developer occasionally deleting his local repository. A tool to purge the local repository the same way Archiva does for its managed repos would be nice, but is not specific to Continuum.
          Hide
          Geert Pante added a comment -

          This is Continuum 1.1.

          As I understand it, continuum uses the 'Deployment Repository Directory' to override the local repository. If it is blank, I guess it uses the current user's default repository (~/.m2/repository). The documentation is not clear for this. Can you explain what you mean by 'The local repository is not optional'?

          I agree that a solution is not specific for Continuum, however using Continuum intensively makes the problem a lot worse.

          Show
          Geert Pante added a comment - This is Continuum 1.1. As I understand it, continuum uses the 'Deployment Repository Directory' to override the local repository. If it is blank, I guess it uses the current user's default repository (~/.m2/repository). The documentation is not clear for this. Can you explain what you mean by 'The local repository is not optional'? I agree that a solution is not specific for Continuum, however using Continuum intensively makes the problem a lot worse.
          Hide
          Wendy Smoak added a comment -

          > As I understand it, continuum uses the 'Deployment Repository Directory' to override the local repository.

          That's not my experience. I find that there is always a local repository (~/.m2/repository, or as configured in settings.xml,) and that the 'deployment repository' is an optional remote repository.

          Show
          Wendy Smoak added a comment - > As I understand it, continuum uses the 'Deployment Repository Directory' to override the local repository. That's not my experience. I find that there is always a local repository (~/.m2/repository, or as configured in settings.xml,) and that the 'deployment repository' is an optional remote repository.
          Hide
          Wendy Smoak added a comment -

          Edited the summary a bit, let's focus on the local repository cleanup with this issue.

          Show
          Wendy Smoak added a comment - Edited the summary a bit, let's focus on the local repository cleanup with this issue.
          Hide
          Baptiste MATHUS added a comment -

          Well, I fear that the dependency:purge-local-repository is not completely what is needed here: this mojo only iterate through the dependencies of the current project pom being used (if I'm not wrong).

          So I guess we'd need a mojo to browse the whole localRepository and delete artefacts based on parameters, like:

          • old snapshots
          • artifact not accessed since x days
          • regexp
            and so on.

          Cheers

          Show
          Baptiste MATHUS added a comment - Well, I fear that the dependency:purge-local-repository is not completely what is needed here: this mojo only iterate through the dependencies of the current project pom being used (if I'm not wrong). So I guess we'd need a mojo to browse the whole localRepository and delete artefacts based on parameters, like: old snapshots artifact not accessed since x days regexp and so on. Cheers
          Hide
          Wendy Smoak added a comment -

          Also consider that there could be more than one local repository.

          You an use -Dmaven.repo.local on the command line to point at a different one, each "Installation" of Maven could have settings.xml configured to default to a different location, etc.

          This may be more of an issue if/when Continuum supports concurrent builds.

          Show
          Wendy Smoak added a comment - Also consider that there could be more than one local repository. You an use -Dmaven.repo.local on the command line to point at a different one, each "Installation" of Maven could have settings.xml configured to default to a different location, etc. This may be more of an issue if/when Continuum supports concurrent builds.
          Hide
          Felix Knecht added a comment -

          Not sure, but looks like your using maven2.

          You can deploy the snapshot file with name 'SNAPSHOT' instead of deploying the snapshot named with timestamp. Thus an already existing snapshot file gets overwritten when deploying the next snapshot.

          See http://maven.apache.org/plugins/maven-deploy-plugin/deploy-file-mojo.html and search for uniqueVersion.
          Maybe it helps.

          Show
          Felix Knecht added a comment - Not sure, but looks like your using maven2. You can deploy the snapshot file with name 'SNAPSHOT' instead of deploying the snapshot named with timestamp. Thus an already existing snapshot file gets overwritten when deploying the next snapshot. See http://maven.apache.org/plugins/maven-deploy-plugin/deploy-file-mojo.html and search for uniqueVersion. Maybe it helps.
          Hide
          Geert Pante added a comment -

          As far as I know, <uniqueVersion> has effect on 'remote' deployment. But our remote repository is archiva, and since 1.0.2, it cleans up old snapshots quite nicely.

          This issue is about keeping the 'local' repository in shape. And according to CONTINUUM-1253, this isn't affected by <uniqueVersion>. CONTINUUM-1253 is, unfortunately, a 'Won't Fix'

          Show
          Geert Pante added a comment - As far as I know, <uniqueVersion> has effect on 'remote' deployment. But our remote repository is archiva, and since 1.0.2, it cleans up old snapshots quite nicely. This issue is about keeping the 'local' repository in shape. And according to CONTINUUM-1253 , this isn't affected by <uniqueVersion>. CONTINUUM-1253 is, unfortunately, a 'Won't Fix'
          Hide
          Wendy Smoak added a comment -

          CONTINUUM-1253 doesn't have anything to do with the local repository. It is related to the deployment repository, use of which is optional.

          At Codehaus Mojo, someone is working on a 'local repository purge' plugin using concepts (if not code) from Archiva: http://www.nabble.com/local-repository-purge-plugin-td16937047.html

          Show
          Wendy Smoak added a comment - CONTINUUM-1253 doesn't have anything to do with the local repository. It is related to the deployment repository, use of which is optional. At Codehaus Mojo, someone is working on a 'local repository purge' plugin using concepts (if not code) from Archiva: http://www.nabble.com/local-repository-purge-plugin-td16937047.html
          Hide
          Marc Lustig added a comment -

          This task is already realized by Archiva.
          Archiva offers options "Repository Purge By Days Older Than" and "Repository Purge By Retention Count".
          And for non-managed repos it's definately a Maven issue.
          I suggest to close this ticket.

          Show
          Marc Lustig added a comment - This task is already realized by Archiva. Archiva offers options "Repository Purge By Days Older Than" and "Repository Purge By Retention Count". And for non-managed repos it's definately a Maven issue. I suggest to close this ticket.
          Hide
          Geert Pante added a comment -

          This is indeed an issue that could be solved by a Maven feature, but if Maven doesn't implement it, Continuum should have a workaround.

          When you use maven in normal development, it is acceptable that your local repositories aren't cleaned up automatically.
          If you use maven in a Continuous Integration environment with a lot of big projects, this is no longer acceptable, and it is a serious threat for large deployments. As a workaround, we have a seperate crontab job to do the cleanup (with 'rm -rf'), and we make sure there is no schedule running at that time.

          So, please do not close this issue without having a proper workaround, or before CONTINUUM-782 is implemented.

          Show
          Geert Pante added a comment - This is indeed an issue that could be solved by a Maven feature, but if Maven doesn't implement it, Continuum should have a workaround. When you use maven in normal development, it is acceptable that your local repositories aren't cleaned up automatically. If you use maven in a Continuous Integration environment with a lot of big projects, this is no longer acceptable, and it is a serious threat for large deployments. As a workaround, we have a seperate crontab job to do the cleanup (with 'rm -rf'), and we make sure there is no schedule running at that time. So, please do not close this issue without having a proper workaround, or before CONTINUUM-782 is implemented.
          Hide
          Olivier Lamy (*$^¨%`£) added a comment -

          fixed with CONTINUUM-782

          Show
          Olivier Lamy (*$^¨%`£) added a comment - fixed with CONTINUUM-782

            People

            • Assignee:
              Olivier Lamy (*$^¨%`£)
              Reporter:
              Geert Pante
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development