[HADOOP-19164] Hadoop CLI MiniCluster is broken

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.5.0, 3.4.2
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Ayush Saxena
• Reporter: Ayush Saxena
• Created: 03/May/24 20:52
• Updated: 23/Sep/24 07:31
• Resolved: 21/Sep/24 15:57

Description

Documentation is also broken & it doesn't work either

(https://apache.github.io/hadoop/hadoop-project-dist/hadoop-common/CLIMiniCluster.html)

Fails with:

Exception in thread "main" java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer
	at org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2666)
	at org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2680)
	at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1510)
	at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:989)
	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:588)
	at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:530)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.start(MiniHadoopClusterManager.java:160)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.run(MiniHadoopClusterManager.java:132)
	at org.apache.hadoop.mapreduce.MiniHadoopClusterManager.main(MiniHadoopClusterManager.java:320)
Caused by: java.lang.ClassNotFoundException: org.mockito.stubbing.Answer
	at java.net.URLClassLoader.findClass(URLClassLoader.java:387)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
	... 9 more

Command executed:

bin/mapred minicluster -format

Documentation Issues:

bin/mapred minicluster -rmport RM_PORT -jhsport JHS_PORT

Without -format option it doesn't work the first time telling Namenode isn't formatted, So, this should be corrected.

2024-05-04 00:35:52,933 WARN namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: NameNode is not formatted.
	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:253)

This isn't required either:

NOTE: You will need protoc 2.5.0 installed.

Issue Links

• is broken by: HDFS-15915 Race condition with async edits logging due to updating txId outside of the namesystem log
• is duplicated by: MAPREDUCE-7492 MiniHadoopClusterManager does not accept writeDetails file path
• relates to: HDFS-16050 Some dynamometer tests fail
• links to: GitHub Pull Request #7050