[MAPREDUCE-7471] Hadoop mapred minicluster command line fails with class not found

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.3.5
• Assignee: Unassigned
• Reporter: Duo Zhang
• Created: 27/Jan/24 12:55
• Updated: 01/Feb/24 10:23

Description

If you run

./bin/mapred minicluster

It will fail with

Exception in thread "Listener at localhost/35325" java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
	at org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2648)
	at org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2662)
	at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1510)
	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:989)
	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:588)
	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:530)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:160)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 9 more

This line

https://github.com/apache/hadoop/blob/835403d872506c4fa76eb2d721f2d91f413473d5/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java#L2648

This is because we rely on mockito in NameNodeAdapter but we do not have mockito on our classpath, at least in our published hadoop-3.3.5 binary.

And there is another problem that, if we do not run the above command in the HADOOP_HOME directory, i.e, in another directory by typing the absolute path of the mapred command, it will fail with

Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
	at org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:336)
	at org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:280)
	at org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:289)
	at org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:3069)
	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.<init>(MiniDFSCluster.java:239)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:157)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
Caused by: java.lang.ClassNotFoundException: org.junit.Assert
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
	... 8 mor

This simply because this line

https://github.com/apache/hadoop/blob/835403d872506c4fa76eb2d721f2d91f413473d5/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh#L601

We should add the $HADOOP_TOOLS_HOME prefix for the default value of HADOOP_TOOLS_DIR.

Issue Links

• relates to: HBASE-28331 Client integration test fails after upgrading hadoop3 version to 3.3.x