[HDFS-12197] Do the HDFS dist stitching in hadoop-hdfs-project

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0-alpha4
• Component/s: build
• Target Version/s: 3.5.0
• Assignee: Unassigned
• Reporter: Andrew Wang
• Created: 25/Jul/17 23:47
• Updated: 04/Jan/24 07:20

Description

Problem reported by lars_francke on HDFS-11596. We can no longer easily start a namenode and datanode from the source directory without doing a full build per the wiki instructions: https://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment

This is because we don't have a top-level dist for HDFS. $HADOOP_YARN_HOME for instance can be set to hadoop-yarn-project/target, but $HADOOP_HDFS_HOME goes into the submodule: hadoop-hdfs-project/hadoop-hdfs/target. This means it's missing the files from the sibling hadoop-hdfs-client module (which is required by the namenode), but also other siblings like nfs and httpfs.

So, I think the right fix is doing the dist stitching at the hadoop-hdfs-project level where we can aggregate all the child modules, and pointing $HADOOP_HDFS_HOME at this directory.

Issue Links

• is broken by: HDFS-11596 hadoop-hdfs-client jar is in the wrong directory in release tarball