[BIGTOP-3893] Fix Hadoop3.3.4 build issues on ppc64le

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 3.2.0
• Assignee: Yuqi Gu
• Reporter: Yuqi Gu
• Created: 15/Dec/22 07:48
• Updated: 30/Jul/23 12:42
• Resolved: 19/Dec/22 02:32

Description

Haoop common depends on protobuf-2.5.0:

[ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.1:compile (src-compile-protoc-legacy) on project hadoop-common: Missing:
[ERROR] ----------
[ERROR] 1) com.google.protobuf:protoc:exe:linux-ppcle_64:2.5.0
[ERROR]
[ERROR]   Try downloading the file manually from the project website.
[ERROR]
[ERROR]   Then, install it using the command:
[ERROR]       mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-ppcle_64 -Dpackaging=exe -Dfile=/path/to/file
[ERROR]
[ERROR]   Alternatively, if you host your own repository you can deploy the file there:
[ERROR]       mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=2.5.0 -Dclassifier=linux-ppcle_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR]   Path to dependency:
[ERROR]         1) org.apache.hadoop:hadoop-common:jar:3.3.4
[ERROR]         2) com.google.protobuf:protoc:exe:linux-ppcle_64:2.5.0
[ERROR]
[ERROR] ----------
[ERROR] 1 required artifact is missing.
[ERROR]

In Haddop YARN CSI, it depends on protobuf-3.6.1 and grpc-java 1.26.0.
But there is no pre-built binary of protobuf-3.6.1 and grpc-java-1.26.0 for ppc64le.
It's needed to build protobuf-3.6.1 and grpc-java Binaries in Bigtop toolchain:

[INFO] Apache Hadoop YARN CSI ............................. FAILURE [  0.552 s

[ERROR] Failed to execute goal org.xolstice.maven.plugins:protobuf-maven-plugin:0.5.1:compile (default) on project hadoop-yarn-csi: Missing:
[ERROR] ----------
[ERROR] 1) com.google.protobuf:protoc:exe:linux-ppcle_64:3.6.1
[ERROR]
[ERROR]   Try downloading the file manually from the project website.
[ERROR]
[ERROR]   Then, install it using the command:
[ERROR]       mvn install:install-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.6.1 -Dclassifier=linux-ppcle_64 -Dpackagin                                                                                                                                                                                       g=exe -Dfile=/path/to/file
[ERROR]
[ERROR]   Alternatively, if you host your own repository you can deploy the file there:
[ERROR]       mvn deploy:deploy-file -DgroupId=com.google.protobuf -DartifactId=protoc -Dversion=3.6.1 -Dclassifier=linux-ppcle_64 -Dpackaging=                                                                                                                                                                                       exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR]   Path to dependency:
[ERROR]         1) org.apache.hadoop:hadoop-yarn-csi:jar:3.3.4
[ERROR]         2) com.google.protobuf:protoc:exe:linux-ppcle_64:3.6.1
[ERROR]
[ERROR] ----------
[ERROR] 1 required artifact is missing.
[ERROR]
[ERROR] for artifact:
[ERROR]   org.apache.hadoop:hadoop-yarn-csi:jar:3.3.4
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR]   apache.snapshots.https (https://repository.apache.org/content/repositories/snapshots, releases=true, snapshots=true),
[ERROR]   repository.jboss.org (https://repository.jboss.org/nexus/content/groups/public/, releases=true, snapshots=false),
[ERROR]   central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)
[ERROR]


#############################################################################
[ERROR] ----------
[ERROR] 1) io.grpc:protoc-gen-grpc-java:exe:linux-ppcle_64:1.26.0
[ERROR]
[ERROR]   Try downloading the file manually from the project website.
[ERROR]
[ERROR]   Then, install it using the command:
[ERROR]       mvn install:install-file -DgroupId=io.grpc -DartifactId=protoc-gen-grpc-java -Dversion=1.26.0 -Dclassifier=linux-ppcle_64 -Dpackaging=exe -Dfile=/path/to/file
[ERROR]
[ERROR]   Alternatively, if you host your own repository you can deploy the file there:
[ERROR]       mvn deploy:deploy-file -DgroupId=io.grpc -DartifactId=protoc-gen-grpc-java -Dversion=1.26.0 -Dclassifier=linux-ppcle_64 -Dpackaging=exe -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR]   Path to dependency:
[ERROR]         1) org.apache.hadoop:hadoop-yarn-csi:jar:3.3.4
[ERROR]         2) io.grpc:protoc-gen-grpc-java:exe:linux-ppcle_64:1.26.0

Issue Links

• relates to: BIGTOP-3970 Fix build failure of Zeppelin due to missing grpc-java-1.26.0 on ppc64le