[BIGTOP-3610] Use a stricter match for the zookeeper jar in hadoop packaging

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Assignee: Luca Toscano
• Reporter: Luca Toscano
• Created: 09/Dec/21 08:00
• Updated: 10/Dec/21 07:07
• Resolved: 10/Dec/21 07:07

Description

The following build failure was found while testing the build of the hadoop package on Debian 10 x86:

BUILD FAILED in 46m 30s
+ ln -fs /usr/lib/hadoop/lib/snappy-java-1.0.5.jar debian/tmp//usr/lib/hadoop/client/snappy-java-1.0.5.jar
+ ln -fs /usr/lib/hadoop/lib/snappy-java-1.0.5.jar debian/tmp//usr/lib/hadoop/client/snappy-java.jar
+ continue 2
+ for file in `cat ${BUILD_DIR}/hadoop-client.list`
+ for dir in ${HADOOP_DIR}/{lib,} ${HDFS_DIR}/{lib,} ${YARN_DIR}/{lib,} ${MAPREDUCE_DIR}/{lib,}
+ '[' -e debian/tmp//usr/lib/hadoop/lib/stax2-api-3.1.4.jar ']'
+ ln -fs /usr/lib/hadoop/lib/stax2-api-3.1.4.jar debian/tmp//usr/lib/hadoop/client/stax2-api-3.1.4.jar
+ ln -fs /usr/lib/hadoop/lib/stax2-api-3.1.4.jar debian/tmp//usr/lib/hadoop/client/stax2-api.jar
+ continue 2
+ for file in `cat ${BUILD_DIR}/hadoop-client.list`
+ for dir in ${HADOOP_DIR}/{lib,} ${HDFS_DIR}/{lib,} ${YARN_DIR}/{lib,} ${MAPREDUCE_DIR}/{lib,}
+ '[' -e debian/tmp//usr/lib/hadoop/lib/token-provider-1.0.1.jar ']'
+ ln -fs /usr/lib/hadoop/lib/token-provider-1.0.1.jar debian/tmp//usr/lib/hadoop/client/token-provider-1.0.1.jar
+ ln -fs /usr/lib/hadoop/lib/token-provider-1.0.1.jar debian/tmp//usr/lib/hadoop/client/token-provider.jar
+ continue 2
+ for file in `cat ${BUILD_DIR}/hadoop-client.list`
+ for dir in ${HADOOP_DIR}/{lib,} ${HDFS_DIR}/{lib,} ${YARN_DIR}/{lib,} ${MAPREDUCE_DIR}/{lib,}
+ '[' -e debian/tmp//usr/lib/hadoop/lib/woodstox-core-5.0.3.jar ']'
+ ln -fs /usr/lib/hadoop/lib/woodstox-core-5.0.3.jar debian/tmp//usr/lib/hadoop/client/woodstox-core-5.0.3.jar
+ ln -fs /usr/lib/hadoop/lib/woodstox-core-5.0.3.jar debian/tmp//usr/lib/hadoop/client/woodstox-core.jar
+ continue 2
# Forcing Zookeeper dependency to be on the packaged jar
ln -sf /usr/lib/zookeeper/zookeeper.jar debian/tmp/usr/lib/hadoop/lib/zookeeper*.jar
ln: target 'debian/tmp/usr/lib/hadoop/lib/zookeeper-jute-3.5.9.jar' is not a directory
make[1]: *** [debian/rules:57: override_dh_auto_install] Error 1
make[1]: Leaving directory '/ws/output/hadoop/hadoop-3.2.2'
make: *** [debian/rules:27: binary] Error 2
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2

The main reason seems to be that there are two jars in the output directory related to zookeper:

$ ls output/hadoop/hadoop-3.2.2/debian/tmp/usr/lib/hadoop/lib/zookeeper-*
output/hadoop/hadoop-3.2.2/debian/tmp/usr/lib/hadoop/lib/zookeeper-3.5.9.jar
output/hadoop/hadoop-3.2.2/debian/tmp/usr/lib/hadoop/lib/zookeeper-jute-3.5.9.jar

The failing command "ln" assumes, in my opinion, that there is only one file matching the zookeeper-* pattern in the output dir, but now we have two. We can make the command smarter but I am wondering if the new jar is needed at runtime (so if needs to be copied/linked as well).

Something like the following seems to work:

diff --git a/bigtop-packages/src/deb/hadoop/rules b/bigtop-packages/src/deb/hadoop/rules
index fe0f3017..e57194b6 100755
--- a/bigtop-packages/src/deb/hadoop/rules
+++ b/bigtop-packages/src/deb/hadoop/rules
@@ -66,7 +66,7 @@ override_dh_auto_install:
          --native-build-string=${native_dir} \
          --installed-lib-dir=/usr/lib/hadoop
        # Forcing Zookeeper dependency to be on the packaged jar
-       ln -sf /usr/lib/zookeeper/zookeeper.jar debian/tmp/usr/lib/hadoop/lib/zookeeper*.jar
+       ln -sf /usr/lib/zookeeper/zookeeper.jar debian/tmp/usr/lib/hadoop/lib/zookeeper-[0-9.]+.jar
        # Workaround for BIGTOP-583
        rm -f debian/tmp/usr/lib/hadoop-*/lib/slf4j-log4j12-*.jar
        # FIXME: BIGTOP-463

But of course it doesn't take into account the jute jar.

The interesting thing is that I can see the following for hadoop.spec as well:

# Forcing Zookeeper dependency to be on the packaged jar
%__ln_s -f /usr/lib/zookeeper/zookeeper.jar $RPM_BUILD_ROOT/%{lib_hadoop}/lib/zookeeper*.jar

Does it happen also for rpm packages? If not, why ?

Issue Links

• links to: GitHub pull request #839