Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1906

Optimize and enahnce build/CI process to reuse artifacts

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0
    • Fix Version/s: backlog
    • Component/s: build
    • Environment:

      all (host as well as any container based building)

      Description

      This was generated from latest "future bigtop" thread on mailing list, capturing core details here from Olaf Flebbe

      cc: Evans Ye

      ==============================================

      However we have a substantial problem hidden deep in th CI „2.0“ approach using containers

      You may know that we place artifacts (i.e. jars) we built with bigtop into the local maven cache ~/.m2. (look for mvn install in do-component-build). The idea is that later maven builds will pick these artifacts and use them rather downloading them from maven central.

      Placing artifacts into ~/.m2 will not have any effect if we use CI containers the way we do now: The maven cache ~/.m2 is lost when the container ends.

      [This triggered misfeature in JIRA https://issues.apache.org/jira/browse/BIGTOP-1893 , BTW: gradle rpm/apt behaved differently from a container build with artifacts from maven central.]

      Option 1) Remove mvn install from all do-component-builds

      Results:

      + We compile projects the way the upstream-developer does.

      • local fixes and configurations will not propagated

      Questions:
      If we do not try to reuse our build-artifacts within compile we have to ask ourself "why do we compile projects at all?“.

      We can build a great test wether someone else has touched / manipulated the maven central cache if we compare artifacts, but is this the really the point of compiling ourselves?

      Option 2) Use mvn install and reuse artifacts even in containers.

      Consequences:

      • Containers are not stateless any more
      • We have to add depencies to CI jobs so they run in order
      • single components may break the whole compile process.
      • Compile does not scale any more

      My Opinion:
      The way we do now "mvn install“ , simply tainting the maven cache seems not a really controlled way to propagate artifacts to me.

      Option 3) Use 1) but reuse artifacts in packages by placing symlinks and dependencies between them.

      • Packages will break with subtile problems if we do symlink artifacts from different releases.

      Neither Option 1, Option 2 nor Option 3 seems a clever way to fix the problem. Would like to hear comments regarding this issue:

      In my humble opinion we should follow Option 2 with all the grave consequences. But maybe reworking mvn install by placing the artifacts with a bigtop specific name / groupid into the maven cache and upload them to maven central .

      Olaf

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                kaiyzen Nate D'Amico
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated: