2018-03-02 17:28:29,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020-inmemoryCompactions-1520011704076] regionserver.MemStoreCompactionStrategy: BASIC info; flattening a segment 2018-03-02 17:28:29,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020-inmemoryCompactions-1520011704076] regionserver.CompactionPipeline: Compaction pipeline segment type=CSLMImmutableSegment, empty=no, cellCount=37399, cellSize=39306349, totalHeapSize=42186288, min timestamp=1520011709407, max timestamp=1520011709754Num uniques -1; flattened 2018-03-02 17:28:29,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020-inmemoryCompactions-1520011704198] regionserver.MemStoreCompactor: Starting on 0453f29030757eedb6e6a1c57e88c085/info 2018-03-02 17:28:29,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020-inmemoryCompactions-1520011704198] regionserver.MemStoreCompactionStrategy: BASIC info; flattening a segment 2018-03-02 17:28:29,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020-inmemoryCompactions-1520011704198] regionserver.CompactionPipeline: Compaction pipeline segment type=CSLMImmutableSegment, empty=no, cellCount=13719, cellSize=14418669, totalHeapSize=15475248, min timestamp=1520011709762, max timestamp=1520011709820Num uniques -1; flattened 2018-03-02 17:28:29,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=16020-inmemoryCompactions-1520011704289] regionserver.MemStoreCompactor: Starting on 0453f29030757eedb6e6a1c57e88c085/info 2018-03-02 17:28:29,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=16020-inmemoryCompactions-1520011704289] regionserver.MemStoreCompactionStrategy: BASIC info; flattening a segment 2018-03-02 17:28:29,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=16020-inmemoryCompactions-1520011704289] regionserver.CompactionPipeline: Compaction pipeline segment type=CSLMImmutableSegment, empty=no, cellCount=13570, cellSize=14262070, totalHeapSize=15307176, min timestamp=1520011709831, max timestamp=1520011709908Num uniques -1; flattened 2018-03-02 17:28:29,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020-inmemoryCompactions-1520011704512] regionserver.MemStoreCompactor: Starting on 0453f29030757eedb6e6a1c57e88c085/info 2018-03-02 17:28:29,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020-inmemoryCompactions-1520011704512] regionserver.MemStoreCompactionStrategy: BASIC info; merging 5 segments 2018-03-02 17:28:30,020 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] regionserver.RSRpcServices: submitting region close type: REGION_NAME value: "TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085." 2018-03-02 17:28:30,020 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] regionserver.RSRpcServices: Close 0453f29030757eedb6e6a1c57e88c085 without moving. Requested from /172.27.28.139 2018-03-02 17:28:30,021 DEBUG [RS_CLOSE_REGION-regionserver/ctr-e138-1518143905142-61260-01-000002:16020-0] handler.CloseRegionHandler: Processing close of TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. 2018-03-02 17:28:30,021 DEBUG [RS_CLOSE_REGION-regionserver/ctr-e138-1518143905142-61260-01-000002:16020-0] regionserver.HRegion: Closing TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.: disabling compactions & flushes 2018-03-02 17:28:30,021 INFO [RS_CLOSE_REGION-regionserver/ctr-e138-1518143905142-61260-01-000002:16020-0] regionserver.HRegion: Running close preflush of TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. 2018-03-02 17:28:30,021 INFO [RS_CLOSE_REGION-regionserver/ctr-e138-1518143905142-61260-01-000002:16020-0] regionserver.HRegion: Flushing 1/1 column families, memstore data size=339.73 MB memstore heap size=349.69 MB 2018-03-02 17:28:30,028 DEBUG [RS_CLOSE_REGION-regionserver/ctr-e138-1518143905142-61260-01-000002:16020-0] regionserver.CompactingMemStore: FLUSHING TO DISK: region TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085.store: info 2018-03-02 17:28:30,028 DEBUG [RS_CLOSE_REGION-regionserver/ctr-e138-1518143905142-61260-01-000002:16020-0] regionserver.CompactionPipeline: Swapping pipeline suffix; before=6, new segment=null 2018-03-02 17:28:30,096 DEBUG [MemStoreFlusher.0] regionserver.HRegion: NOT flushing memstore for region TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085., flushing=false, writesEnabled=false 2018-03-02 17:28:30,096 DEBUG [MemStoreFlusher.0] regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1 2018-03-02 17:28:30,096 DEBUG [MemStoreFlusher.0] regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1 2018-03-02 17:28:30,097 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit: Splitting TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085., compaction_queue=(0:0), split_queue=1 2018-03-02 17:28:30,098 DEBUG [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020] regionserver.IncreasingToUpperBoundRegionSplitPolicy: ShouldSplit because info size=6.9G, sizeToCheck=256.0M, regionsWithCommonTable=1 2018-03-02 17:28:30,296 INFO [RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16020] regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK 2018-03-02 17:28:30,297 DEBUG [MemStoreFlusher.1] regionserver.MemStoreFlusher: Flush thread woke up because memory above low water=381.5 M 2018-03-02 17:28:30,297 INFO [RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16020] regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1] regionserver.MemStoreFlusher: region TestTable,,1520011528142.0453f29030757eedb6e6a1c57e88c085. with size 400432696 2018-03-02 17:28:30,298 DEBUG [MemStoreFlusher.1] regionserver.MemStoreFlusher: region atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. with size 0 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: Flush of region atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. due to global heap pressure. Flush type=ABOVE_ONHEAP_LOWER_MARKTotal Memstore Heap size=381.9 MTotal Memstore Off-Heap size=0, Region memstore size=0 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: wake up by WAKEUPFLUSH_INSTANCE 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: Nothing to flush for atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. 2018-03-02 17:28:30,298 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher: Excluding unflushable region atlas_janus,,1519927429371.fbcb5e495344542daf8b499e4bac03ae. - trying to find a different region to flush. 2018-03-02 17:28:30,298 ERROR [MemStoreFlusher.1] regionserver.MemStoreFlusher: Above memory mark but there are no flushable regions! 2018-03-02 17:28:30,301 INFO [RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16020] regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK 2018-03-02 17:28:30,312 INFO [RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=16020] regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK 2018-03-02 17:28:30,327 INFO [RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=16020] regionserver.MemStoreFlusher: wake up flusher due to ABOVE_ONHEAP_LOWER_MARK