[HBASE-23861] Reconcile Hadoop version

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1
• Fix Version/s: 3.0.0-alpha-1, 2.3.0
• Component/s: dependencies

Environment

Apache Maven 3.6.1
Hadoop 3.2.0 and above.

• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 17/Feb/20 17:02
• Updated: 07/Apr/20 21:37
• Resolved: 18/Mar/20 17:01

Description

I followed the HBase book http://hbase.apache.org/book.html#maven.build.hadoop and wanted to build HBase (master) on top of Hadoop 3.2/3.3 but tests failed right away.

Build:

mvn clean install -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1 -DskipTests

Test:

mvn test -Dtest=TestHelloHBase -Dhadoop.profile=3.0 -Dhadoop-three.version=3.2.1

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.296 s <<< FAILURE! - in org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase
[ERROR] org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase  Time elapsed: 1.284 s  <<< ERROR!
java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/protocol/HdfsConstants$StoragePolicySatisfierMode
at org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase.beforeClass(TestHelloHBase.java:54)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocol.HdfsConstants$StoragePolicySatisfierMode
at org.apache.hbase.archetypes.exemplars.shaded_client.TestHelloHBase.beforeClass(TestHelloHBase.java:54)

Adding mvn -X parameter, I was able to tell that it was because the hbase-server module includes hadoop-distcp and hadoop-dfs-client 3.1.2 (the default Hadoop 3 dependency version) while it uses version 3.2.1 of other hadoop jars . The classpath conflict (the storage policy satisfier is a new feature in Hadoop 3.2) failed the test.

This is reproducible on any Hadoop version 3.2 and above. It looks to me the version of hadoop-distcp and hadoop-hdfs-client should be specified at the top level pom (they are specified in hbase-server/pom.xml).

Issue Links

• is related to: HBASE-22953 Supporting Hadoop 3.3.0
• links to: GitHub Pull Request #1179
• links to: GitHub Pull Request #1242