[BIGTOP-1906] Optimize and enahnce build/CI process to reuse artifacts

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.0.0
• Fix Version/s: backlog
• Component/s: build
• Labels: build containers maven

Environment

all (host as well as any container based building)

• Assignee: Unassigned
• Reporter: Nate D'Amico
• Created: 19/Jun/15 19:42
• Updated: 22/Jan/16 21:13

Description

This was generated from latest "future bigtop" thread on mailing list, capturing core details here from oflebbe

cc: evans_ye

==============================================

However we have a substantial problem hidden deep in th CI „2.0“ approach using containers

You may know that we place artifacts (i.e. jars) we built with bigtop into the local maven cache ~/.m2. (look for mvn install in do-component-build). The idea is that later maven builds will pick these artifacts and use them rather downloading them from maven central.

Placing artifacts into ~/.m2 will not have any effect if we use CI containers the way we do now: The maven cache ~/.m2 is lost when the container ends.

[This triggered misfeature in JIRA https://issues.apache.org/jira/browse/BIGTOP-1893 , BTW: gradle rpm/apt behaved differently from a container build with artifacts from maven central.]

Option 1) Remove mvn install from all do-component-builds

Results:

+ We compile projects the way the upstream-developer does.

• local fixes and configurations will not propagated

Questions:
If we do not try to reuse our build-artifacts within compile we have to ask ourself "why do we compile projects at all?“.

We can build a great test wether someone else has touched / manipulated the maven central cache if we compare artifacts, but is this the really the point of compiling ourselves?

Option 2) Use mvn install and reuse artifacts even in containers.

Consequences:

• Containers are not stateless any more

• We have to add depencies to CI jobs so they run in order

• single components may break the whole compile process.

• Compile does not scale any more

My Opinion:
The way we do now "mvn install“ , simply tainting the maven cache seems not a really controlled way to propagate artifacts to me.

Option 3) Use 1) but reuse artifacts in packages by placing symlinks and dependencies between them.

• Packages will break with subtile problems if we do symlink artifacts from different releases.

---

Neither Option 1, Option 2 nor Option 3 seems a clever way to fix the problem. Would like to hear comments regarding this issue:

In my humble opinion we should follow Option 2 with all the grave consequences. But maybe reworking mvn install by placing the artifacts with a bigtop specific name / groupid into the maven cache and upload them to maven central .

Olaf

Issue Links

• is related to: BIGTOP-2025 Make BOM to be a directional graph