[HDFS-15792] ClasscastException while loading FSImage

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.1, 3.4.0, 2.10.2
• Component/s: nn
• Hadoop Flags: Reviewed
• Assignee: Renukaprasad C
• Reporter: Renukaprasad C
• Created: 27/Jan/21 06:41
• Updated: 22/Mar/21 03:15
• Resolved: 07/Feb/21 10:43

Description

FSImage loading has failed with ClasscastException - java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode.

This is the usage issue with Hashmap in concurrent scenarios.

Same issue has been reported on Java & closed as usage issue. - https://bugs.openjdk.java.net/browse/JDK-8173671

2020-12-28 11:36:26,127 | ERROR | main | An exception occurred when loading INODE from fsiamge. | FSImageFormatProtobuf.java:442
java.lang.
: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode
at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1835)
at java.util.HashMap$TreeNode.treeify(HashMap.java:1951)
at java.util.HashMap.treeifyBin(HashMap.java:772)
at java.util.HashMap.putVal(HashMap.java:644)
at java.util.HashMap.put(HashMap.java:612)
at org.apache.hadoop.hdfs.util.ReferenceCountMap.put(ReferenceCountMap.java:53)
at org.apache.hadoop.hdfs.server.namenode.AclStorage.addAclFeature(AclStorage.java:391)
at org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields.addAclFeature(INodeWithAdditionalFields.java:349)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectory(FSImageFormatPBINode.java:225)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINode(FSImageFormatPBINode.java:406)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.readPBINodes(FSImageFormatPBINode.java:367)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeSection(FSImageFormatPBINode.java:342)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader$2.call(FSImageFormatProtobuf.java:469)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2020-12-28 11:36:26,130 | ERROR | main | Failed to load image from FSImageFile(file=/srv/BigData/namenode/current/fsimage_0000000000198227480, cpktTxId=0000000000198227480) | FSImage.java:738
java.io.IOException: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode
at org.apache.hadoop.io.MultipleIOException$Builder.add(MultipleIOException.java:68)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.runLoaderTasks(FSImageFormatProtobuf.java:444)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:360)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:263)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:227)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:971)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:955)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:820)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:733)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:331)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1113)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:730)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:648)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:710)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:953)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:926)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1665)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1735)
Caused by: java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode
at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1835)
at java.util.HashMap$TreeNode.treeify(HashMap.java:1951)
at java.util.HashMap.treeifyBin(HashMap.java:772)
at java.util.HashMap.putVal(HashMap.java:644)
at java.util.HashMap.put(HashMap.java:612)
at org.apache.hadoop.hdfs.util.ReferenceCountMap.put(ReferenceCountMap.java:53)
at org.apache.hadoop.hdfs.server.namenode.AclStorage.addAclFeature(AclStorage.java:391)
at org.apache.hadoop.hdfs.server.namenode.INodeWithAdditionalFields.addAclFeature(INodeWithAdditionalFields.java:349)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeDirectory(FSImageFormatPBINode.java:225)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINode(FSImageFormatPBINode.java:406)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.readPBINodes(FSImageFormatPBINode.java:367)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Loader.loadINodeSection(FSImageFormatPBINode.java:342)
at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader$2.call(FSImageFormatProtobuf.java:469)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Issue Links

• is related to: HDFS-15907 Reduce Memory Overhead of AclFeature by avoiding AtomicInteger
• relates to: HDFS-14617 Improve fsimage load time by writing sub-sections to the fsimage index