[HBASE-1992] We give up reporting splits if .META. is offline or unreachable (handlers at their maximum)

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Assignee: Unassigned
• Reporter: Michael Stack
• Created: 18/Nov/09 23:31
• Updated: 11/Jun/22 23:15
• Resolved: 21/Apr/11 06:35

Description

Here is our fail reporting a split. Lets make sure that this kinda event is not dropped as part of master rewrite

2009-11-17 20:57:51,943 INFO org.apache.hadoop.hbase.regionserver.HRegion: region dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767/459329844 available; sequence id is 25187748
2009-11-17 20:57:51,944 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767
2009-11-17 20:57:52,007 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 'dds-test,j\xFD\xD7e\x8B\x1
0\xFC\xFE,1258446041463', STARTKEY => 'j\xFD\xD7e\x8B\x10\xFC\xFE', ENDKEY => 'k\x27q6\xCA\xC8\x15,', ENCODED => 1069248233, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 'dds-test', FAMILIES => [{NAME =
> 'data', COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME => 'meta', COMPRESSION => 'NONE', VERSIONS => '1', TTL => '21474
83647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions: dds-test,j\xFD\xD7e\x8B\x10\xFC\xFE,1258520203767, dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767. Split took 1min
s, 8sec
2009-11-17 20:57:52,007 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505
2009-11-17 20:57:54,779 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767
2009-11-17 20:57:54,780 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767
2009-11-17 20:57:54,864 INFO org.apache.hadoop.hbase.regionserver.HRegion: region dds-test,k\x12\xB3\xA6\x02\xE9H\xDC,1258520203767/459329844 available; sequence id is 25187748
2009-11-17 20:58:16,166 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 6 on 60020 took 2967ms appending an edit to hlog; editcount=58
2009-11-17 20:58:16,166 WARN org.apache.hadoop.hbase.regionserver.HLog: regionserver/10.10.0.180:60020.logFlusher took 2854ms optional sync'ing hlog; editcount=59
2009-11-17 20:58:34,340 WARN org.apache.hadoop.hbase.regionserver.HLog: IPC Server handler 6 on 60020 took 7615ms appending an edit to hlog; editcount=820
2009-11-17 20:58:34,340 WARN org.apache.hadoop.hbase.regionserver.HLog: regionserver/10.10.0.180:60020.logFlusher took 1019ms optional sync'ing hlog; editcount=821
2009-11-17 21:00:19,366 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505 in 2mins, 27sec
2009-11-17 21:00:19,367 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505
2009-11-17 21:00:22,237 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505
2009-11-17 21:00:32,829 INFO org.apache.hadoop.hbase.regionserver.HRegion: region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258520419408/538348405 available; sequence id is 25188602
2009-11-17 21:00:32,829 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258520419408
2009-11-17 21:00:33,002 INFO org.apache.hadoop.hbase.regionserver.HRegion: region dds-test,\x7C\xAD\xCF\x27\x93\x10\x9A5,1258520419408/1604966836 available; sequence id is 25188603
2009-11-17 21:00:33,003 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed dds-test,\x7C\xAD\xCF\x27\x93\x10\x9A5,1258520419408
2009-11-17 21:01:51,211 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction/Split failed for region dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server 10.10.0.189:60020 for region .META.,,1, row 'dds-test,\x7C\x98\xB4\x91\xE7L\xDC\xC7,1258436623505', but failed after 1
0 attempts.
Exceptions:
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused
java.net.ConnectException: Connection refused

        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1001)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.doCall(HConnectionManager.java:1192)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1114)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1200)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:605)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:470)
        at org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplitThread.java:211)
        at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:106)
2009-11-17 21:02:00,709 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region dds-test,d\xC7s\x1B\xD3\x23we,1258519740878
2009-11-17 21:02:11,147 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region dds-test,d\xC7s\x1B\xD3\x23we,1258519740878 in 10sec
2009-11-17 21:02:11,147 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region dds-test,a\x96\x85\xB6fV\xE1\xFB,1258519760278
2009-11-17 21:02:30,950 INFO org.apache.hadoop.hbase.regionserver.HLog: Roll /hbase/.logs/sl0127,60020,1258512074419/hlog.dat.1258520252642, entries=853, calcsize=1428860, filesize=1342144. New hlog /hbase/.l
ogs/sl0127,60020,1258512074419/hlog.dat.1258520550914
2009-11-17 21:02:30,950 WARN org.apache.hadoop.hbase.regionserver.HLog: regionserver/10.10.0.180:60020.logFlusher took 7413ms optional sync'ing hlog; editcount=0
2009-11-17 21:03:29,990 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region dds-test,a\x96\x85\xB6fV\xE1\xFB,1258519760278 in 1mins, 18sec

Issue Links

• blocks: HBASE-1816 Master rewrite
• is related to: HBASE-2182 rpc/ipc refactor; x-version compatibility, nio, async, enveloping, sane timeouts, etc.