2019-07-29 20:54:52,225 INFO [RpcServer.priority.FPBQ.Fifo.handler=16,queue=0,port=16020] regionserver.RSRpcServices: Open day_dns,\xE3\xB7\x00\x00,1559807270491.88c2f0c13772132256517d197303a01d. 2019-07-29 20:54:52,227 INFO [RpcServer.priority.FPBQ.Fifo.handler=16,queue=0,port=16020] regionserver.RSRpcServices: Open hour_http,\xC1\xCE\x80\xD4,1561596998827.a8f6d33f234d9da2e775237e869dbe7f. 2019-07-29 20:54:52,229 INFO [RpcServer.priority.FPBQ.Fifo.handler=16,queue=0,port=16020] regionserver.RSRpcServices: Open sni_view,candiapharm.guidechem.com,1563398927797.1ff0f6cbd8c7b7fc0cef6c3c642ae733. 2019-07-29 20:54:52,230 INFO [RpcServer.priority.FPBQ.Fifo.handler=16,queue=0,port=16020] regionserver.RSRpcServices: Open http_view,www.thebeehivefm,1558047442286.4b31beb2c2a0c7f94fd240c5e821807c. 2019-07-29 20:54:52,232 INFO [RpcServer.priority.FPBQ.Fifo.handler=16,queue=0,port=16020] regionserver.RSRpcServices: Open cert_view,d2aee20a3c5dc9797611e5ed5e6686a9,1543765029884.b53978397b09670107bd55d29500015f. 2019-07-29 20:54:52,279 INFO [split-log-closeStream-1] wal.WALSplitter: Rename hdfs://cluster1/hbase/default/hour_cert/3e66c20e3c5a0f86aa40b47a8dc2fc59/recovered.edits/0000000000000786254-dn4%2C16020%2C1564404572638.1564404620761.temp to hdfs://cluster1/hbase/default/hour_cert/3e66c20e3c5a0f86aa40b47a8dc2fc59/recovered.edits/0000000000000786254 2019-07-29 20:54:52,279 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-0] wal.WALSplitter: Processed 1 edits across 1 regions; edits skipped=12; log file=hdfs://cluster1/hbase/WALs/dn4,16020,1564404572638-splitting/dn4%2C16020%2C1564404572638.1564404620761, length=6209, corrupted=false, progress failed=false 2019-07-29 20:54:52,281 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-0] coordination.ZkSplitLogWorkerCoordination: successfully transitioned task /hbase/splitWAL/WALs%2Fdn4%2C16020%2C1564404572638-splitting%2Fdn4%252C16020%252C1564404572638.1564404620761 to final state DONE dn1,16020,1564404602086 2019-07-29 20:54:52,281 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-0] handler.WALSplitterHandler: Worker dn1,16020,1564404602086 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@4376ea54 in 489ms. Status = DONE 2019-07-29 20:54:52,324 INFO [SplitLogWorker-dn1:16020] coordination.ZkSplitLogWorkerCoordination: worker dn1,16020,1564404602086 acquired task /hbase/splitWAL/WALs%2Fh2%2C16020%2C1564216856546-splitting%2Fh2%252C16020%252C1564216856546.1564398538121 2019-07-29 20:54:52,336 INFO [RS_OPEN_REGION-regionserver/dn1:16020-2] regionserver.DefaultStoreFlusher: Flushed memstore data size=37.13 KB at sequenceid=59298272 (bloomFilter=true), to=hdfs://cluster1/hbase/data/default/hour_cert/18fbe2395ced46ec7b0ce0f7a227d187/.tmp/a/94ebcdae8ab54f49b9fe159710e1658e 2019-07-29 20:54:52,341 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] wal.WALSplitter: Splitting WAL=hdfs://cluster1/hbase/WALs/h2,16020,1564216856546-splitting/h2%2C16020%2C1564216856546.1564398538121, length=615233 2019-07-29 20:54:52,341 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] util.FSHDFSUtils: Recover lease on dfs file hdfs://cluster1/hbase/WALs/h2,16020,1564216856546-splitting/h2%2C16020%2C1564216856546.1564398538121 2019-07-29 20:54:52,342 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] util.FSHDFSUtils: Recovered lease, attempt=0 on file=hdfs://cluster1/hbase/WALs/h2,16020,1564216856546-splitting/h2%2C16020%2C1564216856546.1564398538121 after 1ms 2019-07-29 20:54:52,347 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] wal.WALSplitter: 3 split writers finished; closing. 2019-07-29 20:54:52,347 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] wal.WALSplitter: Processed 0 edits across 0 regions; edits skipped=0; log file=hdfs://cluster1/hbase/WALs/h2,16020,1564216856546-splitting/h2%2C16020%2C1564216856546.1564398538121, length=615233, corrupted=false, progress failed=false 2019-07-29 20:54:52,347 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] handler.WALSplitterHandler: Worker dn1,16020,1564404602086 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@75737e16 in 23ms. Status = null 2019-07-29 20:54:52,348 ERROR [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] executor.EventHandler: Caught throwable while processing event RS_LOG_REPLAY java.lang.ArrayIndexOutOfBoundsException: 16403 at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365) at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358) at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:735) at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:143) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEdit(WALEdit.java:148) at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297) at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:195) at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:100) at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2019-07-29 20:54:52,352 ERROR [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] regionserver.HRegionServer: ***** ABORTING region server dn1,16020,1564404602086: Caught throwable while processing event RS_LOG_REPLAY ***** java.lang.ArrayIndexOutOfBoundsException: 16403 at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1365) at org.apache.hadoop.hbase.KeyValue.getFamilyLength(KeyValue.java:1358) at org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(PrivateCellUtil.java:735) at org.apache.hadoop.hbase.CellUtil.matchingFamily(CellUtil.java:816) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEditFamily(WALEdit.java:143) at org.apache.hadoop.hbase.wal.WALEdit.isMetaEdit(WALEdit.java:148) at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297) at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:195) at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:100) at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:70) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2019-07-29 20:54:52,353 ERROR [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2019-07-29 20:54:52,364 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] regionserver.HRegionServer: Dump of metrics as JSON on abort: { "beans" : [ { "name" : "java.lang:type=Memory", "modelerType" : "sun.management.MemoryImpl", "ObjectPendingFinalizationCount" : 0, "NonHeapMemoryUsage" : { "committed" : 90439680, "init" : 2555904, "max" : -1, "used" : 88133952 }, "Verbose" : true, "HeapMemoryUsage" : { "committed" : 53687091200, "init" : 53687091200, "max" : 53687091200, "used" : 676914400 }, "ObjectName" : "java.lang:type=Memory" } ], "beans" : [ ], "beans" : [ ], "beans" : [ ] } 2019-07-29 20:54:52,371 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] regionserver.HRegionServer: ***** STOPPING region server 'dn1,16020,1564404602086' ***** 2019-07-29 20:54:52,371 INFO [RS_LOG_REPLAY_OPS-regionserver/dn1:16020-1] regionserver.HRegionServer: STOPPED: Caught throwable while processing event RS_LOG_REPLAY 2019-07-29 20:54:52,372 INFO [regionserver/dn1:16020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread 2019-07-29 20:54:52,372 INFO [regionserver/dn1:16020] regionserver.HRegionServer: Stopping infoServer 2019-07-29 20:54:52,372 WARN [SplitLogWorker-dn1:16020] coordination.ZkSplitLogWorkerCoordination: Interrupted while yielding for other region servers java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.grabTask(ZkSplitLogWorkerCoordination.java:267) at org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.taskLoop(ZkSplitLogWorkerCoordination.java:441) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:144) at java.lang.Thread.run(Thread.java:745) 2019-07-29 20:54:52,372 INFO [SplitLogWorker-dn1:16020] regionserver.SplitLogWorker: SplitLogWorker dn1,16020,1564404602086 exiting 2019-07-29 20:54:52,375 INFO [RS_OPEN_REGION-regionserver/dn1:16020-2] regionserver.HStore: Added hdfs://cluster1/hbase/data/default/hour_cert/18fbe2395ced46ec7b0ce0f7a227d187/a/94ebcdae8ab54f49b9fe159710e1658e, entries=341, sequenceid=59298272, filesize=44.1 K 2019-07-29 20:54:52,375 INFO [RS_OPEN_REGION-regionserver/dn1:16020-2] regionserver.HRegion: Finished flush of dataSize ~37.13 KB/38025, heapSize ~73.61 KB/75376, currentSize=0 B/0 for 18fbe2395ced46ec7b0ce0f7a227d187 in 523ms, sequenceid=59298272, compaction requested=true; wal=null 2019-07-29 20:54:52,381 INFO [regionserver/dn1:16020] handler.ContextHandler: Stopped o.e.j.w.WebAppContext@21579b35{/,null,UNAVAILABLE}{file:/opt/dns_query/hbase-2.1.1/hbase-webapps/regionserver} 2019-07-29 20:54:52,385 INFO [regionserver/dn1:16020] server.AbstractConnector: Stopped ServerConnector@55574295{HTTP/1.1,[http/1.1]}{0.0.0.0:16030} 2019-07-29 20:54:52,386 INFO [regionserver/dn1:16020] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@7d43f1c9{/static,file:///opt/dns_query/hbase-2.1.1/hbase-webapps/static/,UNAVAILABLE} 2019-07-29 20:54:52,386 INFO [regionserver/dn1:16020] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@3ec62141{/logs,file:///opt/dns_query/hbase-2.1.1/logs/,UNAVAILABLE} 2019-07-29 20:54:52,387 INFO [main-BucketCacheWriter-1] bucket.BucketCache: main-BucketCacheWriter-1 exiting, cacheEnabled=false 2019-07-29 20:54:52,387 INFO [main-BucketCacheWriter-0] bucket.BucketCache: main-BucketCacheWriter-0 exiting, cacheEnabled=false 2019-07-29 20:54:52,387 INFO [main-BucketCacheWriter-2] bucket.BucketCache: main-BucketCacheWriter-2 exiting, cacheEnabled=false 2019-07-29 20:54:52,396 INFO [regionserver/dn1:16020] bucket.BucketCache: Shutdown bucket cache: IO persistent=false; path to write=null 2019-07-29 20:54:52,396 INFO [regionserver/dn1:16020] regionserver.HeapMemoryManager: Stopping 2019-07-29 20:54:52,397 INFO [regionserver/dn1:16020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly. 2019-07-29 20:54:52,397 INFO [regionserver/dn1:16020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly. 2019-07-29 20:54:52,397 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting 2019-07-29 20:54:52,397 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting 2019-07-29 20:54:52,399 INFO [regionserver/dn1:16020] regionserver.HRegionServer: aborting server dn1,16020,1564404602086 2019-07-29 20:54:52,399 INFO [regionserver/dn1:16020] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x0177e812 to dn1:2181,dn2:2181,dn3:2181,nna:2181,nns:2181 2019-07-29 20:54:52,400 INFO [regionserver/dn1:16020] regionserver.CompactSplit: Waiting for Split Thread to finish... 2019-07-29 20:54:52,404 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-0] regionserver.HRegion: Closed http_view,weihai.64824.com,1558047464412.06354884ee77e2ea30293757562c1eea. 2019-07-29 20:54:52,404 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-1] regionserver.HRegion: Closed hour_dns,\xCBw,\xB7,1542143112327.c2fbc7c1103cc1071a0020db8bff359b. 2019-07-29 20:54:52,404 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-2] regionserver.HRegion: Closed day_dns,\xED\xF3\x00\x00,1559807270491.88ed0de16cf5a3ab6258bb2c08ed5818. 2019-07-29 20:54:52,407 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-0] regionserver.HRegion: Closed hour_sni,\xDD\xED{A,1559187935415.fe1cbf10a1cc9b8e780e66e491b35f9c. 2019-07-29 20:54:52,407 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-1] regionserver.HRegion: Closed hour_sni,\xC6m\xE9;143.204.174.37\x02,1561190437746.76b4e6ec15c795a7dc33eb12709b29ab. 2019-07-29 20:54:52,408 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-2] regionserver.HRegion: Closed hour_cert,\xA0\x03\xC4R18aa7693faefe9c344b0ccadf9c24ff2\x01,1564376912002.2c742128df1a9a2e5f4e6cb7ddc3b92f. 2019-07-29 20:54:52,409 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-0] regionserver.HRegion: Closed day_dns,\xCF?\x00\x00,1559807270491.869841e88ecf8a417d98f33e34fcd202. 2019-07-29 20:54:52,409 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-2] regionserver.HRegion: Closed http_view,yingkou.sdshsk.com,1557617818753.bd8fc27d5c751b6c65c3b0c9a05b5c69. 2019-07-29 20:54:52,409 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-1] regionserver.HRegion: Closed hour_http,\xEDWU;xlxerzgh6.dxcpic.cn\x01,1560801780546.d48957a5303aa343d53481146f2a4661. 2019-07-29 20:54:52,410 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-1] regionserver.HRegion: Closed cert_view_v2,ff\x00\x00,1559279839692.f7ac7c0f0a669ad52b01db1450001264. 2019-07-29 20:54:52,410 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-2] regionserver.HRegion: Closed dns_view,www.pyrenees-chambresdhotes.com,1550881435739.edfb5f8afaad4d82314086096b21550c. 2019-07-29 20:54:52,411 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-1] regionserver.HRegion: Closed cert_view_v2,,1559280322083.df1e2ac570e30c2897b4e2471ee1e1c0. 2019-07-29 20:54:52,411 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-2] regionserver.HRegion: Closed dns_link_v3,yz8v8kib.jjj.com\x091\x092\x0969.171.245.5,1554768341333.f5501574076aa1e135b36965a0614b1a. 2019-07-29 20:54:52,519 INFO [RS_OPEN_REGION-regionserver/dn1:16020-2] regionserver.HRegion: Opened 18fbe2395ced46ec7b0ce0f7a227d187; next sequenceid=59298273 2019-07-29 20:54:52,559 INFO [RS_OPEN_REGION-regionserver/dn1:16020-2] regionserver.HRegion: Closed hour_cert,\x99\x96\x00\x00,1564376912002.18fbe2395ced46ec7b0ce0f7a227d187. 2019-07-29 20:54:52,613 INFO [StoreOpener-8c46ceb90c30855579517947dc44fa28-1] regionserver.HStore: Store=a, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2019-07-29 20:54:52,668 INFO [RS_OPEN_REGION-regionserver/dn1:16020-0] regionserver.HRegion: Opened 8c46ceb90c30855579517947dc44fa28; next sequenceid=1918568 2019-07-29 20:54:52,702 INFO [RS_OPEN_REGION-regionserver/dn1:16020-0] regionserver.HRegion: Closed sni_view,sztxsjcl.chinawj.com.cn,1543613653845.8c46ceb90c30855579517947dc44fa28. 2019-07-29 20:54:53,085 INFO [RS_OPEN_REGION-regionserver/dn1:16020-1] regionserver.DefaultStoreFlusher: Flushed memstore data size=36.69 MB at sequenceid=70050128 (bloomFilter=true), to=hdfs://cluster1/hbase/data/default/cert_view/2ef063b663800f89e9c1a75a185f24ce/.tmp/a/208f30552c7e438394452fca6779083d 2019-07-29 20:54:53,145 INFO [RS_OPEN_REGION-regionserver/dn1:16020-1] regionserver.HStore: Added hdfs://cluster1/hbase/data/default/cert_view/2ef063b663800f89e9c1a75a185f24ce/a/208f30552c7e438394452fca6779083d, entries=237957, sequenceid=70050128, filesize=37.9 M 2019-07-29 20:54:53,145 INFO [RS_OPEN_REGION-regionserver/dn1:16020-1] regionserver.HRegion: Finished flush of dataSize ~36.69 MB/38475558, heapSize ~61.48 MB/64469888, currentSize=0 B/0 for 2ef063b663800f89e9c1a75a185f24ce in 960ms, sequenceid=70050128, compaction requested=true; wal=null 2019-07-29 20:54:53,273 INFO [RS_OPEN_REGION-regionserver/dn1:16020-1] regionserver.HRegion: Opened 2ef063b663800f89e9c1a75a185f24ce; next sequenceid=70050129 2019-07-29 20:54:53,314 INFO [RS_OPEN_REGION-regionserver/dn1:16020-1] regionserver.HRegion: Closed cert_view,f8ac3202fa95ea628a70687aa93637b4,1543755068898.2ef063b663800f89e9c1a75a185f24ce. 2019-07-29 20:54:53,393 INFO [regionserver/dn1:16020.leaseChecker] regionserver.Leases: Closed leases 2019-07-29 20:54:53,439 INFO [regionserver/dn1:16020.Chore.1] hbase.ScheduledChore: Chore: CompactedHFilesCleaner was stopped 2019-07-29 20:54:53,439 INFO [regionserver/dn1:16020.Chore.1] hbase.ScheduledChore: Chore: CompactionChecker was stopped 2019-07-29 20:54:53,439 INFO [regionserver/dn1:16020.Chore.1] hbase.ScheduledChore: Chore: MemstoreFlusherChore was stopped 2019-07-29 20:54:53,439 INFO [regionserver/dn1:16020.Chore.1] hbase.ScheduledChore: Chore: MovedRegionsCleaner for region dn1,16020,1564404602086 was stopped 2019-07-29 20:54:53,684 INFO [regionserver/dn1:16020-longCompactions-1564404653349] throttle.PressureAwareThroughputController: 82632245fed6823f747a04364a71a3aa#a#compaction#2 average throughput is 6.60 MB/second, slept 2 time(s) and total slept time is 2921 ms. 1 active operations remaining, total limit is 11.13 MB/second 2019-07-29 20:54:53,719 INFO [regionserver/dn1:16020-longCompactions-1564404653349] regionserver.HRegion: compaction interrupted java.io.InterruptedIOException: Aborting compaction of store a in region hour_http,\x90\x09\xFD\x1352.24.70.82\x02,1560947928355.82632245fed6823f747a04364a71a3aa. because it was interrupted. at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:331) at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65) at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126) at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2163) at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:592) at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:634) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2019-07-29 20:54:53,720 INFO [regionserver/dn1:16020-longCompactions-1564404653349] regionserver.CompactSplit: Aborted compaction region=hour_http,\x90\x09\xFD\x1352.24.70.82\x02,1560947928355.82632245fed6823f747a04364a71a3aa., storeName=a, priority=92, startTime=1564404889161; duration=4sec 2019-07-29 20:54:53,720 INFO [regionserver/dn1:16020] regionserver.CompactSplit: Waiting for Large Compaction Thread to finish... 2019-07-29 20:54:53,722 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-1] regionserver.HRegion: Closed hour_http,\x90\x09\xFD\x1352.24.70.82\x02,1560947928355.82632245fed6823f747a04364a71a3aa. 2019-07-29 20:54:53,754 INFO [regionserver/dn1:16020-shortCompactions-1564404888981] throttle.PressureAwareThroughputController: 5873480392dfb9ed0e0d9327325095c3#a#compaction#1 average throughput is 6.49 MB/second, slept 2 time(s) and total slept time is 2722 ms. 0 active operations remaining, total limit is 11.13 MB/second 2019-07-29 20:54:53,782 INFO [regionserver/dn1:16020-shortCompactions-1564404888981] regionserver.HRegion: compaction interrupted java.io.InterruptedIOException: Aborting compaction of store a in region sni_link_v2,151.101.130.66\x092\x091\x09b6fc,1541972498560.5873480392dfb9ed0e0d9327325095c3. because it was interrupted. at org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:331) at org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:65) at org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:126) at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1407) at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2163) at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:592) at org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:634) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2019-07-29 20:54:53,782 INFO [regionserver/dn1:16020-shortCompactions-1564404888981] regionserver.CompactSplit: Aborted compaction region=sni_link_v2,151.101.130.66\x092\x091\x09b6fc,1541972498560.5873480392dfb9ed0e0d9327325095c3., storeName=a, priority=86, startTime=1564404888981; duration=4sec 2019-07-29 20:54:53,782 INFO [regionserver/dn1:16020] regionserver.CompactSplit: Waiting for Small Compaction Thread to finish... 2019-07-29 20:54:53,783 INFO [regionserver/dn1:16020] regionserver.HRegionServer: Waiting on 2 regions to close 2019-07-29 20:54:53,783 INFO [RS_CLOSE_REGION-regionserver/dn1:16020-0] regionserver.HRegion: Closed sni_link_v2,151.101.130.66\x092\x091\x09b6fc,1541972498560.5873480392dfb9ed0e0d9327325095c3. 2019-07-29 20:54:53,784 ERROR [RS_CLOSE_META-regionserver/dn1:16020-0] regionserver.HRegion: Memstore data size is 78305 2019-07-29 20:54:53,785 INFO [RS_CLOSE_META-regionserver/dn1:16020-0] regionserver.HRegion: Closed hbase:meta,,1.1588230740 2019-07-29 20:54:53,983 INFO [regionserver/dn1:16020] regionserver.HRegionServer: stopping server dn1,16020,1564404602086; all regions closed. 2019-07-29 20:54:54,045 INFO [regionserver/dn1:16020] regionserver.Leases: Closed leases 2019-07-29 20:54:54,046 INFO [regionserver/dn1:16020] hbase.ChoreService: Chore service for: regionserver/dn1:16020 had [[ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS]] on shutdown 2019-07-29 20:54:54,046 INFO [regionserver/dn1:16020.logRoller] regionserver.LogRoller: LogRoller exiting. 2019-07-29 20:54:54,047 INFO [regionserver/dn1:16020] ipc.NettyRpcServer: Stopping server on /11.172.43.46:16020 2019-07-29 20:54:54,061 INFO [regionserver/dn1:16020] zookeeper.ZooKeeper: Session: 0x5000018dc790220 closed 2019-07-29 20:54:54,061 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x5000018dc790220 2019-07-29 20:54:54,061 INFO [regionserver/dn1:16020] regionserver.HRegionServer: Exiting; stopping=dn1,16020,1564404602086; zookeeper connection closed. 2019-07-29 20:54:54,061 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting java.lang.RuntimeException: HRegionServer Aborted at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:67) at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:3021) 2019-07-29 20:54:54,062 INFO [Thread-4] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@5016f1a 2019-07-29 20:54:54,063 INFO [Thread-4] regionserver.ShutdownHook: Starting fs shutdown hook thread. 2019-07-29 20:54:54,063 INFO [Thread-4] regionserver.ShutdownHook: Shutdown hook finished.