Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-1992

We give up reporting splits if .META. is offline or unreachable (handlers at their maximum)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Here is our fail reporting a split. Lets make sure that this kinda event is not dropped as part of master rewrite

      2009-11-17 20:57:51,943 INFO org.apache.hadoop.hbase.regionserver.HRegion: region dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767/459329844 available; sequence id is 25187748
      2009-11-17 20:57:51,944 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767
      2009-11-17 20:57:52,007 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'dds-test,j\xFD\xD7e\x8B\x1
      0\xFC\xFE,1258446041463', STARTKEY => 'j\xFD\xD7e\x8B\x10\xFC\xFE', ENDKEY => 'k\x27q6\xCA\xC8\x15,', ENCODED => 1069248233, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'dds-test', FAMILIES => [{NAME =
      > 'data', COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'meta', COMPRESSION => 'NONE', VERSIONS => '1', TTL => '21474
      83647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions: dds-test,j\xFD\xD7e\x8B\x10\xFC\xFE,1258520203767, dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767. Split took 1min
      s, 8sec
      2009-11-17 20:57:52,007 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505
      2009-11-17 20:57:54,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767
      2009-11-17 20:57:54,780 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767
      2009-11-17 20:57:54,864 INFO org.apache.hadoop.hbase.regionserver.HRegion: region dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767/459329844 available; sequence id is 25187748
      2009-11-17 20:58:16,166 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 6 on 60020 took 2967ms appending an edit to hlog; editcount=58
      2009-11-17 20:58:16,166 WARN org.apache.hadoop.hbase.regionserver.HLog: regionserver/10.10.0.180:60020.logFlusher took 2854ms optional sync'ing hlog; editcount=59
      2009-11-17 20:58:34,340 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 6 on 60020 took 7615ms appending an edit to hlog; editcount=820
      2009-11-17 20:58:34,340 WARN org.apache.hadoop.hbase.regionserver.HLog: regionserver/10.10.0.180:60020.logFlusher took 1019ms optional sync'ing hlog; editcount=821
      2009-11-17 21:00:19,366 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505 in 2mins, 27sec
      2009-11-17 21:00:19,367 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505
      2009-11-17 21:00:22,237 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505
      2009-11-17 21:00:32,829 INFO org.apache.hadoop.hbase.regionserver.HRegion: region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258520419408/538348405 available; sequence id is 25188602
      2009-11-17 21:00:32,829 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258520419408
      2009-11-17 21:00:33,002 INFO org.apache.hadoop.hbase.regionserver.HRegion: region dds-test,\x7C\xAD\xCF\x27\x93\x10\x9A5,1258520419408/1604966836 available; sequence id is 25188603
      2009-11-17 21:00:33,003 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed dds-test,\x7C\xAD\xCF\x27\x93\x10\x9A5,1258520419408
      2009-11-17 21:01:51,211 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction/Split failed for region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505
      org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.10.0.189:60020 for region .META.,,1, row 'dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505', but failed after 1
      0 attempts.
      Exceptions:
      java.net.ConnectException: Connection refused
      java.net.ConnectException: Connection refused
      java.net.ConnectException: Connection refused
      java.net.ConnectException: Connection refused
      java.net.ConnectException: Connection refused
      java.net.ConnectException: Connection refused
      java.net.ConnectException: Connection refused
      java.net.ConnectException: Connection refused
      java.net.ConnectException: Connection refused
      java.net.ConnectException: Connection refused
      
              at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1001)
              at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.doCall(HConnectionManager.java:1192)
              at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1114)
              at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1200)
              at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:605)
              at org.apache.hadoop.hbase.client.HTable.put(HTable.java:470)
              at org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:211)
              at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:106)
      2009-11-17 21:02:00,709 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region dds-test,d\xC7s\x1B\xD3\x23we,1258519740878
      2009-11-17 21:02:11,147 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region dds-test,d\xC7s\x1B\xD3\x23we,1258519740878 in 10sec
      2009-11-17 21:02:11,147 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region dds-test,a\x96\x85\xB6fV\xE1\xFB,1258519760278
      2009-11-17 21:02:30,950 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll /hbase/.logs/sl0127,60020,1258512074419/hlog.dat.1258520252642, entries=853, calcsize=1428860, filesize=1342144. New hlog /hbase/.l
      ogs/sl0127,60020,1258512074419/hlog.dat.1258520550914
      2009-11-17 21:02:30,950 WARN org.apache.hadoop.hbase.regionserver.HLog: regionserver/10.10.0.180:60020.logFlusher took 7413ms optional sync'ing hlog; editcount=0
      2009-11-17 21:03:29,990 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region dds-test,a\x96\x85\xB6fV\xE1\xFB,1258519760278 in 1mins, 18sec
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stack Michael Stack
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: