Details
-
Task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
ghx-label-8
Description
The hadoop binaries that we download from the CDP build number are built for x86_64. On x86_64, HADOOP_LIB_DIR and HADOOP_INCLUDE_DIR point to the CDP hadoop (i.e. HADOOP_HOME/lib and HADOOP_HOME/include). Various pieces (including the C++ build) use these environment variables to find the native libraries.
On ARM, we leave those environment variables pointed to that same location. We fix things up by downloading a separate hadoop-client built for ARM, then copying the contents into the usual location in the CDP hadoop directory, overwriting the x86_64 contents. The code to overwrite the libraries runs on each invocation of buildall.sh
On ARM, we could change this to point HADOOP_LIB_DIR to the downloaded hadoop-client (which is built for ARM). With a bit of work on the hadoop-client, we could get it to also have the header files and also point HADOOP_INCLUDE_DIR to it. This avoids the need to copy files during buildall.sh. Any build that wants to pass in a custom hadoop can then use HADOOP_LIB_DIR_OVERRIDE and HADOOP_INCLUDE_DIR_OVERRIDE.