Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-543 A region's state is kept in several places in the master opening the possibility for race conditions
  3. HBASE-1046

Region assigned to two regionservers after split

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • None
    • 0.19.0
    • regionserver
    • None

    Description

      After reassignment of region split, scanners (mapred tasks) fail:

      org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.30.94.49:60020 for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, row 'c4e518dc579ca98bf97ea13d1f7c80fc', but failed after 10 attempts.
      Exceptions:
      java.io.IOException: java.io.IOException: HStoreScanner failed construction
      at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:70)
      at org.apache.hadoop.hbase.regionserver.HStoreScanner.(HStoreScanner.java:84)
      at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2113)
      at org.apache.hadoop.hbase.regionserver.HRegion$HScanner.(HRegion.java:1847)
      at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1158)
      at org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:1521)
      at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
      Caused by: java.io.FileNotFoundException: File does not exist: hdfs://sjdc-atr-dc-1.atr.trendmicro.com:50000/data/hbase/content/1137747785/info/mapfiles/3405780294391796198/data
      at org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:394)
      at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:695)
      at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1420)
      at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1415)
      at org.apache.hadoop.io.MapFile$Reader.createDataFileReader(MapFile.java:301)
      at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.createDataFileReader(HBaseMapFile.java:98)
      at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:283)
      at org.apache.hadoop.hbase.io.HBaseMapFile$HBaseReader.(HBaseMapFile.java:81)
      at org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.(BloomFilterMapFile.java:66)
      at org.apache.hadoop.hbase.io.HalfMapFileReader.(HalfMapFileReader.java:86)
      at org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:438)
      at org.apache.hadoop.hbase.regionserver.StoreFileScanner.openReaders(StoreFileScanner.java:96)
      at org.apache.hadoop.hbase.regionserver.StoreFileScanner.(StoreFileScanner.java:67)
      ... 10 more

      According to the master the regions around this row are:

      content,c4580f7292ceb2f6c8d732d935302633,1228328680554 10.30.94.50:60020 693011073 c4580f7292ceb2f6c8d732d935302633 c4e518dc579ca98bf97ea13d1f7c80fc
      content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 10.30.94.49:60020 2031068422 c4e518dc579ca98bf97ea13d1f7c80fc c528c8864e67a3addcb50587586dfe38

      On 10.30.94.49, just before the HStoreScanner exceptions begin:

      2008-12-04 16:18:30,538 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
      2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
      2008-12-04 16:18:30,539 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c528c8864e67a3addcb50587586dfe38,1228407504257
      2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
      2008-12-04 16:18:30,633 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
      2008-12-04 16:18:30,725 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
      2008-12-04 16:18:36,659 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
      2008-12-04 16:18:36,673 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c528c8864e67a3addcb50587586dfe38,1228407504257
      java.lang.NullPointerException
      at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
      at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
      at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
      at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)
      2008-12-04 16:18:36,689 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
      2008-12-04 16:18:36,691 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
      java.lang.NullPointerException
      at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:863)
      at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:708)
      at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:666)
      at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:87)

      This is the region history:

      content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
      Thu, 4 Dec 2008 11:18:33 open Region opened on server : 10.30.94.49
      Thu, 4 Dec 2008 11:18:30 assignment Region assigned to server 10.30.94.49:60020
      Thu, 4 Dec 2008 11:18:28 open Region opened on server : 10.30.94.54:60020
      Thu, 4 Dec 2008 11:18:25 assignment Region assigned to server 10.30.94.54:60020
      Thu, 4 Dec 2008 11:18:25 split Region split from: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554

      On 10.30.94.49:

      2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
      MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
      2008-12-04 16:17:39,550 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
      MSG_REGION_OPEN: content,61938ad11c92493c1aac82932651b873,1228317826934
      2008-12-04 16:18:17,469 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
      2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: compac
      tion completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554 in 6sec
      2008-12-04 16:18:24,255 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
      2008-12-04 16:18:24,258 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554
      2008-12-04 16:18:24,745 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
      2008-12-04 16:18:24,746 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
      2008-12-04 16:18:25,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c528c8864e67a3addcb50587586dfe38,1228407504257/1813242175 available
      2008-12-04 16:18:25,052 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed content,c528c8864e67a3addcb50587586dfe38,1228407504257
      2008-12-04 16:18:25,066 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=
      REGION => {NAME => content,c4e518dc579ca98bf97ea13d1f7c80fc,1228328680554', STARTKEY => 'c4e518dc579ca98bf97ea13d1f7c80fc', ENDKEY => c56eb508be0009d04429b6977c
      91a425', ENCODED => 1137747785, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'content', IS_ROOT => 'false', IS_META => 'false', FAMILIES => [

      {NAME => 'url', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}

      ,

      {NAME => 'info', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}

      ,

      {NAME => 'content', BLOOMFILTER => 'false', VERSIONS => '1', COMPRESSION => 'NONE', LENGTH => '2147483647', TTL => '2592000', IN_MEMORY => 'false', BLOCKCACHE => 'false'}

      ]}}, new regions: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257, content,c528c8864e67a3addcb50587586dfe38,1228407504257. Split took 0sec

      On 10.30.94.54:

      2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegionServer:
      Worker: MSG_REGION_OPEN: content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
      2008-12-04 16:18:25,681 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c528c8864e67a3addcb50587586dfe38,1228407504257
      2008-12-04 16:18:25,867 INFO org.apache.hadoop.hbase.regionserver.HRegion: region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257/2031068422 available
      2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c528c8864e67a3addcb50587586dfe38,1228407504257 in 7sec
      2008-12-04 16:18:32,951 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257
      2008-12-04 16:18:35,882 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region content,c4e518dc579ca98bf97ea13d1f7c80fc,1228407504257 in 2sec

      Attachments

        Activity

          People

            jimk Jim Kellerman
            apurtell Andrew Kyle Purtell
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: