[HBASE-1810] ConcurrentModificationException in region assignment

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.20.0
• Fix Version/s: 0.20.0
• Hadoop Flags: Reviewed
• Assignee: Mathias Herberts
• Reporter: Lars George
• Created: 02/Sep/09 10:39
• Updated: 13/Sep/09 22:24
• Resolved: 02/Sep/09 16:40

Description

2009-09-01 11:28:24,106 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60000, call regionServerReport(address: 192.168.105.39:60020, startcode: 1251828463043, load: (requests=0, regions=21, usedHeap=135, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@6556c280, [Lorg.apache.hadoop.hbase.HRegionInfo;@22fb957a) from 192.168.105.39:60281: error: java.io.IOException: java.util.ConcurrentModificationException
java.io.IOException: java.util.ConcurrentModificationException
        at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
        at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
        at org.apache.hadoop.hbase.master.RegionManager.isMetaServer(RegionManager.java:837)
        at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:405)
        at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)

and

2009-09-01 11:28:25,713 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60000, call regionServerReport(address: 192.168.105.37:60020, startcode: 1251828463132, load: (requests=0, regions=14, usedHeap=141, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@68345260, [Lorg.apache.hadoop.hbase.HRegionInfo;@430c5212) from 192.168.105.37:35432: error: java.io.IOException: java.util.ConcurrentModificationException
java.io.IOException: java.util.ConcurrentModificationException
        at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
        at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
        at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:428)
        at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
        at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
        at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)

As discussed with Mathias on IRC:

<hbs> regionsInTransition is a synchronized map so accesses to it are synchronized but not looping over it
<hbs> as stated in http://java.sun.com/j2se/1.5.0/docs/api/java/util/Collections.html#synchronizedMap%28java.util.Map%29
<hbs> so there is a missing synchronized(regionsInTransition) wherever regionsInTransition is iterated over.