Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-1810

ConcurrentModificationException in region assignment

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      2009-09-01 11:28:24,106 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60000, call regionServerReport(address: 192.168.105.39:60020, startcode: 1251828463043, load: (requests=0, regions=21, usedHeap=135, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@6556c280, [Lorg.apache.hadoop.hbase.HRegionInfo;@22fb957a) from 192.168.105.39:60281: error: java.io.IOException: java.util.ConcurrentModificationException
      java.io.IOException: java.util.ConcurrentModificationException
              at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
              at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
              at org.apache.hadoop.hbase.master.RegionManager.isMetaServer(RegionManager.java:837)
              at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:405)
              at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
              at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
              at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
              at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
              at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
              at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
              at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
      

      and

      2009-09-01 11:28:25,713 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60000, call regionServerReport(address: 192.168.105.37:60020, startcode: 1251828463132, load: (requests=0, regions=14, usedHeap=141, maxHeap=4093), [Lorg.apache.hadoop.hbase.HMsg;@68345260, [Lorg.apache.hadoop.hbase.HRegionInfo;@430c5212) from 192.168.105.37:35432: error: java.io.IOException: java.util.ConcurrentModificationException
      java.io.IOException: java.util.ConcurrentModificationException
              at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1100)
              at java.util.TreeMap$ValueIterator.next(TreeMap.java:1145)
              at org.apache.hadoop.hbase.master.RegionManager.regionsAwaitingAssignment(RegionManager.java:428)
              at org.apache.hadoop.hbase.master.RegionManager.assignRegions(RegionManager.java:202)
              at org.apache.hadoop.hbase.master.ServerManager.processMsgs(ServerManager.java:481)
              at org.apache.hadoop.hbase.master.ServerManager.processRegionServerAllsWell(ServerManager.java:415)
              at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:324)
              at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:722)
              at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
              at java.lang.reflect.Method.invoke(Method.java:597)
              at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:650)
              at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
      

      As discussed with Mathias on IRC:

      <hbs> regionsInTransition is a synchronized map so accesses to it are synchronized but not looping over it
      <hbs> as stated in http://java.sun.com/j2se/1.5.0/docs/api/java/util/Collections.html#synchronizedMap%28java.util.Map%29
      <hbs> so there is a missing synchronized(regionsInTransition) wherever regionsInTransition is iterated over.

        Attachments

        1. HBASE-1810.patch
          2 kB
          Mathias Herberts

          Activity

            People

            • Assignee:
              herberts Mathias Herberts
              Reporter:
              larsgeorge Lars George
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: