[HADOOP-18220] Improve build performance of modules

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: build
• Assignee: Unassigned
• Reporter: Bing-ok
• Created: 03/May/22 17:12
• Updated: 03/May/22 17:12

Description

Hi, I'm now using hadoop. I found that the build time of the project is not very fast when I used  mvn -T 1C install -DskipTests command to build the project during Github Actions. So I try to speed up the build performance of the project. The goal here is to clean up dependencies between maven modules in the project in order to improve the build performance.

Thread usage before cleaning up module dependencies was this:

Using mvn dependency:analyze command I managed to have an overview of the unused dependencies of every module. I focused only on the dependencies between modules. The following dependencies can be cleaned up:

org.apache.hadoop:hadoop-client-minicluster(hadoop-client-modules/hadoop-client-minicluster) -> org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime)
org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime) -> org.apache.hadoop:hadoop-client-api(hadoop-client-modules/hadoop-client-api)
org.apache.hadoop:hadoop-client-runtime(hadoop-client-modules/hadoop-client-runtime) -> org.apache.hadoop:hadoop-client(hadoop-client-modules/hadoop-client)
org.apache.hadoop:hadoop-dist(hadoop-dist) -> org.apache.hadoop:hadoop-client-check-test-invariants(hadoop-client-modules/hadoop-client-check-test-invariants)

After clean up, the build time has been reduced by 3min when I use parallel build during Github Actions.
Thread usage after cleaning up module dependencies was this:

Could you help me review this issue? I can submit a PR to improve build performance of the project.

Thank you very much for your attention.
Best regards.