Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1906

Optimize and enahnce build/CI process to reuse artifacts



    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: backlog
    • Component/s: build
    • Environment:

      all (host as well as any container based building)


      This was generated from latest "future bigtop" thread on mailing list, capturing core details here from Olaf Flebbe

      cc: Evans Ye


      However we have a substantial problem hidden deep in th CI „2.0“ approach using containers

      You may know that we place artifacts (i.e. jars) we built with bigtop into the local maven cache ~/.m2. (look for mvn install in do-component-build). The idea is that later maven builds will pick these artifacts and use them rather downloading them from maven central.

      Placing artifacts into ~/.m2 will not have any effect if we use CI containers the way we do now: The maven cache ~/.m2 is lost when the container ends.

      [This triggered misfeature in JIRA https://issues.apache.org/jira/browse/BIGTOP-1893 , BTW: gradle rpm/apt behaved differently from a container build with artifacts from maven central.]

      Option 1) Remove mvn install from all do-component-builds


      + We compile projects the way the upstream-developer does.

      • local fixes and configurations will not propagated

      If we do not try to reuse our build-artifacts within compile we have to ask ourself "why do we compile projects at all?“.

      We can build a great test wether someone else has touched / manipulated the maven central cache if we compare artifacts, but is this the really the point of compiling ourselves?

      Option 2) Use mvn install and reuse artifacts even in containers.


      • Containers are not stateless any more
      • We have to add depencies to CI jobs so they run in order
      • single components may break the whole compile process.
      • Compile does not scale any more

      My Opinion:
      The way we do now "mvn install“ , simply tainting the maven cache seems not a really controlled way to propagate artifacts to me.

      Option 3) Use 1) but reuse artifacts in packages by placing symlinks and dependencies between them.

      • Packages will break with subtile problems if we do symlink artifacts from different releases.

      Neither Option 1, Option 2 nor Option 3 seems a clever way to fix the problem. Would like to hear comments regarding this issue:

      In my humble opinion we should follow Option 2 with all the grave consequences. But maybe reworking mvn install by placing the artifacts with a bigtop specific name / groupid into the maven cache and upload them to maven central .



          Issue Links



              • Assignee:
                kaiyzen Nate D'Amico
              • Votes:
                0 Vote for this issue
                8 Start watching this issue


                • Created: