[HBASE-25717] RegionServer aborted due to ClassCastException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.6, 2.3.4, 2.4.2
• Fix Version/s: 3.0.0-alpha-1, 2.1.10, 2.2.7, 2.5.0, 2.4.3, 2.3.6
• Hadoop Flags: Reviewed
• Assignee: Pankaj Kumar
• Reporter: chiranjeevi
• Created: 30/Mar/21 09:14
• Updated: 15/Apr/21 22:07
• Resolved: 08/Apr/21 11:05

Description

Observed RegionServer crash while writing huge data concurrently through YCSB tool,

2021-03-29 14:35:37,860 | ERROR | RpcServer.priority.RWQ.Fifo.write.handler=19,queue=3,port=RS_PORT | ***** ABORTING region server RS_IP,RS_PORT,1616999325078: The coprocessor org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode ***** | org.slf4j.helpers.MarkerIgnoringBase.error(MarkerIgnoringBase.java:159)
java.lang.ClassCastException: java.util.HashMap$Node cannot be cast to java.util.HashMap$TreeNode
at java.util.HashMap$TreeNode.moveRootToFront(HashMap.java:1835)
at java.util.HashMap$TreeNode.treeify(HashMap.java:1951)
at java.util.HashMap.treeifyBin(HashMap.java:772)
at java.util.HashMap.putVal(HashMap.java:644)
at java.util.HashMap.put(HashMap.java:612)
at java.util.HashSet.add(HashSet.java:220)
at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.registerAndMarkMeter(MetaTableMetrics.java:200)
at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.regionMetricRegisterAndMark(MetaTableMetrics.java:173)
at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.registerAndMarkMetrics(MetaTableMetrics.java:106)
at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.prePut(MetaTableMetrics.java:91)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$23.call(RegionCoprocessorHost.java:948)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$23.call(RegionCoprocessorHost.java:945)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:581)
at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:655)
at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.prePut(RegionCoprocessorHost.java:945)

This problem occured only once in my test cluster. Discussed offline with pankajkumar it looks like race condition in MetaTableMetrics.metrics.

Issue Links

• links to: GitHub Pull Request #3108