2018-12-13 13:52:41,847 INFO [MemStoreFlusher.2] regionserver.HRegion: Started memstore flush for realtime_dw:HB_RT_FDM_AFS_CSP_APPEND_ORDER_CHAIN,5ffffffa,1537426822457.e5d1feb58e576ac621acf0c14c2a574e., current region memstore size 4.36 MB, and 1/1 column families' memstores are being flushed. 2018-12-13 13:52:41,871 INFO [MemStoreFlusher.2] regionserver.DefaultStoreFlusher: Flushed, sequenceid=125456743, memsize=4.4 M, hasBloomFilter=true, into tmp file hdfs://pallas/hbase/data/realtime_dw/HB_RT_FDM_AFS_CSP_APPEND_ORDER_CHAIN/e5d1feb58e576ac621acf0c14c2a574e/.tmp/4b162259f3ac4a89b3b31df7568909f1 2018-12-13 13:52:41,879 INFO [MemStoreFlusher.2] regionserver.HStore: Added hdfs://pallas/hbase/data/realtime_dw/HB_RT_FDM_AFS_CSP_APPEND_ORDER_CHAIN/e5d1feb58e576ac621acf0c14c2a574e/d/4b162259f3ac4a89b3b31df7568909f1, entries=962, sequenceid=125456743, filesize=26.5 K 2018-12-13 13:52:41,880 INFO [MemStoreFlusher.2] regionserver.HRegion: Finished memstore flush of ~4.36 MB/4576512, currentsize=0 B/0 for region realtime_dw:HB_RT_FDM_AFS_CSP_APPEND_ORDER_CHAIN,5ffffffa,1537426822457.e5d1feb58e576ac621acf0c14c2a574e. in 33ms, sequenceid=125456743, compaction requested=false 2018-12-13 13:54:04,006 INFO [HT-HBASE-PALLAS-11-7-73-235.hadoop.jd.local,16020,1544603147118_ChoreService_1] regionserver.HRegionServer: HT-HBASE-PALLAS-11-7-73-235.hadoop.jd.local,16020,1544603147118-MemstoreFlusherChore requesting flush for region realtime_table:HB_RT_FDM_PRODUCT_POP_SKU_CHAIN,2664817b43ccb2cf63c211dd2e897f47_24031915857,1535128777277.4b9996cc151ddb6d6dd38c892fa2b82b. after a delay of 14869 2018-12-13 13:54:05,077 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: #3, waiting for 1 actions to finish on table: realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN 2018-12-13 13:54:05,077 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Left over 1 task(s) are processed on server(s): [HT-HBASE-PALLAS-11-7-79-70.hadoop.jd.local,16020,1544603147058, HT-HBASE-PALLAS-11-7-113-76.hadoop.jd.local,16020,1544603147230] 2018-12-13 13:54:05,087 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Regions against which left over task(s) are processed: [keeper_ns:message,jd_510d2d237c85e_0423165b8e,1542624080825.664af35614b603bf08d258454365fd0d., realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN,266616173970c3763836bb5a626fc453_11899118897,1543092690573.3bf4b6ad59e03a6f2d2e5f778138ea29.] 2018-12-13 13:54:14,006 INFO [HT-HBASE-PALLAS-11-7-73-235.hadoop.jd.local,16020,1544603147118_ChoreService_1] regionserver.HRegionServer: HT-HBASE-PALLAS-11-7-73-235.hadoop.jd.local,16020,1544603147118-MemstoreFlusherChore requesting flush for region realtime_table:HB_RT_FDM_PRODUCT_POP_SKU_CHAIN,2664817b43ccb2cf63c211dd2e897f47_24031915857,1535128777277.4b9996cc151ddb6d6dd38c892fa2b82b. after a delay of 21716 2018-12-13 13:54:14,006 INFO [HT-HBASE-PALLAS-11-7-73-235.hadoop.jd.local,16020,1544603147118_ChoreService_1] regionserver.HRegionServer: HT-HBASE-PALLAS-11-7-73-235.hadoop.jd.local,16020,1544603147118-MemstoreFlusherChore requesting flush for region realtime_table:HB_INDEX_FDM_POP_SELLER_S32_0_VENDER_OPERATOR_CHAIN,b3333328,1540816255339.0bcd294ceab744c02a121b571427d921. after a delay of 13302 2018-12-13 13:54:15,082 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: #3, waiting for 1 actions to finish on table: realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN 2018-12-13 13:54:15,082 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Left over 1 task(s) are processed on server(s): [HT-HBASE-PALLAS-11-7-73-69.hadoop.jd.local,16020,1544603147324, HT-HBASE-PALLAS-11-7-113-76.hadoop.jd.local,16020,1544603147230] 2018-12-13 13:54:15,093 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Regions against which left over task(s) are processed: [athena_device_data:metrics,,1542219050264.8e6e169c72760927d87e5572b4bb0b52., realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN,266616173970c3763836bb5a626fc453_11899118897,1543092690573.3bf4b6ad59e03a6f2d2e5f778138ea29., tmlog_ns:span,ee_6cacbf0a0c16df43,1542507309123.e23d01248bbadf0d2bab1f914dc1fc79.] 2018-12-13 13:54:15,284 INFO [LoadIncrementalHFiles-0] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@7146064e, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-13 13:54:15,284 INFO [LoadIncrementalHFiles-1] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@7146064e, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-13 13:54:15,286 INFO [LoadIncrementalHFiles-0] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://pallas/user/hadp/hbase-staging/hadp__fbi_fbi_user_indicator__hf2al343ntdtihcaauiuv4odt429b15c973iek50nf7kelh7aru1algjjv7fpdcs/d/37c1e220dea1410a91a941a340b2946d_SeqId_80000295_ first=eccd1f92*yx_481118924 last=ed396fc6jd_6cbc801ee756b 2018-12-13 13:54:15,286 INFO [LoadIncrementalHFiles-1] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://pallas/user/hadp/hbase-staging/hadp__fbi_fbi_user_indicator__hf2al343ntdtihcaauiuv4odt429b15c973iek50nf7kelh7aru1algjjv7fpdcs/d/71121cf13c984ca0a59e2bf6cab5bfea_SeqId_80000295_ first=f18b6b3a\xE7\x89\x9F\xE7\x81\xBE\xE6\x8B\x93\xE5\x9F\x94\xE6\x93\x85\xE6\x85\x9596 last=f333a288lanv524945_m 2018-12-13 13:54:15,286 INFO [LoadIncrementalHFiles-1] mapreduce.LoadIncrementalHFiles: HFile at hdfs://pallas/user/hadp/hbase-staging/hadp__fbi_fbi_user_indicator__hf2al343ntdtihcaauiuv4odt429b15c973iek50nf7kelh7aru1algjjv7fpdcs/d/71121cf13c984ca0a59e2bf6cab5bfea_SeqId_80000295_ no longer fits inside a single region. Splitting... 2018-12-13 13:54:15,287 INFO [LoadIncrementalHFiles-1] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@7146064e, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-13 13:54:15,302 INFO [LoadIncrementalHFiles-1] hfile.CacheConfig: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@7146064e, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2018-12-13 13:54:18,875 INFO [MemStoreFlusher.2] regionserver.HRegion: Started memstore flush for realtime_table:HB_RT_FDM_PRODUCT_POP_SKU_CHAIN,2664817b43ccb2cf63c211dd2e897f47_24031915857,1535128777277.4b9996cc151ddb6d6dd38c892fa2b82b., current region memstore size 185.66 MB, and 1/1 column families' memstores are being flushed. 2018-12-13 13:54:19,987 INFO [MemStoreFlusher.2] regionserver.DefaultStoreFlusher: Flushed, sequenceid=700769356, memsize=185.7 M, hasBloomFilter=true, into tmp file hdfs://pallas/hbase/data/realtime_table/HB_RT_FDM_PRODUCT_POP_SKU_CHAIN/4b9996cc151ddb6d6dd38c892fa2b82b/.tmp/360ae40a881d437da2690ee1af465720 2018-12-13 13:54:19,995 INFO [MemStoreFlusher.2] regionserver.HStore: Added hdfs://pallas/hbase/data/realtime_table/HB_RT_FDM_PRODUCT_POP_SKU_CHAIN/4b9996cc151ddb6d6dd38c892fa2b82b/d/360ae40a881d437da2690ee1af465720, entries=549159, sequenceid=700769356, filesize=12.2 M 2018-12-13 13:54:19,996 INFO [MemStoreFlusher.2] regionserver.HRegion: Finished memstore flush of ~185.66 MB/194679816, currentsize=0 B/0 for region realtime_table:HB_RT_FDM_PRODUCT_POP_SKU_CHAIN,2664817b43ccb2cf63c211dd2e897f47_24031915857,1535128777277.4b9996cc151ddb6d6dd38c892fa2b82b. in 1121ms, sequenceid=700769356, compaction requested=false 2018-12-13 13:54:24,006 INFO [HT-HBASE-PALLAS-11-7-73-235.hadoop.jd.local,16020,1544603147118_ChoreService_1] regionserver.HRegionServer: HT-HBASE-PALLAS-11-7-73-235.hadoop.jd.local,16020,1544603147118-MemstoreFlusherChore requesting flush for region realtime_table:HB_INDEX_FDM_POP_SELLER_S32_0_VENDER_OPERATOR_CHAIN,b3333328,1540816255339.0bcd294ceab744c02a121b571427d921. after a delay of 14393 2018-12-13 13:54:25,091 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: #3, waiting for 1 actions to finish on table: realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN 2018-12-13 13:54:25,092 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Left over 1 task(s) are processed on server(s): [HT-HBASE-PALLAS-11-7-113-76.hadoop.jd.local,16020,1544603147230] 2018-12-13 13:54:25,097 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Regions against which left over task(s) are processed: [realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN,266616173970c3763836bb5a626fc453_11899118897,1543092690573.3bf4b6ad59e03a6f2d2e5f778138ea29.] 2018-12-13 13:54:27,308 INFO [MemStoreFlusher.2] regionserver.HRegion: Started memstore flush for realtime_table:HB_INDEX_FDM_POP_SELLER_S32_0_VENDER_OPERATOR_CHAIN,b3333328,1540816255339.0bcd294ceab744c02a121b571427d921., current region memstore size 248 B, and 1/1 column families' memstores are being flushed. 2018-12-13 13:54:27,317 INFO [MemStoreFlusher.2] regionserver.DefaultStoreFlusher: Flushed, sequenceid=120005248, memsize=248, hasBloomFilter=true, into tmp file hdfs://pallas/hbase/data/realtime_table/HB_INDEX_FDM_POP_SELLER_S32_0_VENDER_OPERATOR_CHAIN/0bcd294ceab744c02a121b571427d921/.tmp/c98641ddb1c1497aa50092f4ecca0411 2018-12-13 13:54:27,324 INFO [MemStoreFlusher.2] regionserver.HStore: Added hdfs://pallas/hbase/data/realtime_table/HB_INDEX_FDM_POP_SELLER_S32_0_VENDER_OPERATOR_CHAIN/0bcd294ceab744c02a121b571427d921/d/c98641ddb1c1497aa50092f4ecca0411, entries=1, sequenceid=120005248, filesize=5.0 K 2018-12-13 13:54:27,325 INFO [MemStoreFlusher.2] regionserver.HRegion: Finished memstore flush of ~248 B/248, currentsize=0 B/0 for region realtime_table:HB_INDEX_FDM_POP_SELLER_S32_0_VENDER_OPERATOR_CHAIN,b3333328,1540816255339.0bcd294ceab744c02a121b571427d921. in 17ms, sequenceid=120005248, compaction requested=false 2018-12-13 13:54:55,112 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: #3, waiting for 1 actions to finish on table: realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN 2018-12-13 13:54:55,112 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Left over 1 task(s) are processed on server(s): [HT-HBASE-PALLAS-11-7-113-76.hadoop.jd.local,16020,1544603147230] 2018-12-13 13:54:55,119 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Regions against which left over task(s) are processed: [realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN,266616173970c3763836bb5a626fc453_11899118897,1543092690573.3bf4b6ad59e03a6f2d2e5f778138ea29.] 2018-12-13 13:55:05,121 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: #3, waiting for 1 actions to finish on table: realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN 2018-12-13 13:55:05,121 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Left over 1 task(s) are processed on server(s): [HT-HBASE-PALLAS-11-7-113-76.hadoop.jd.local,16020,1544603147230] 2018-12-13 13:55:05,128 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Regions against which left over task(s) are processed: [realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN,266616173970c3763836bb5a626fc453_11899118897,1543092690573.3bf4b6ad59e03a6f2d2e5f778138ea29.] 2018-12-13 13:55:06,403 INFO [pool-22-thread-118] regionserver.ReplicationSinkManager: Current list of sinks is out of date or empty, updating 2018-12-13 13:55:06,403 INFO [pool-22-thread-115] regionserver.ReplicationSinkManager: Current list of sinks is out of date or empty, updating 2018-12-13 13:55:11,875 INFO [MemStoreFlusher.3] regionserver.HRegion: Started memstore flush for realtime_dw:HB_RT_INDEX_FDM_PAIMAI_PAIMAI_ORDER_2_CHAIN,33333333,1540362902687.3f5f05afc2ae5cb5c6062245d2453ef4., current region memstore size 2.58 KB, and 1/1 column families' memstores are being flushed. 2018-12-13 13:55:11,884 INFO [MemStoreFlusher.3] regionserver.DefaultStoreFlusher: Flushed, sequenceid=40209714, memsize=2.6 K, hasBloomFilter=true, into tmp file hdfs://pallas/hbase/data/realtime_dw/HB_RT_INDEX_FDM_PAIMAI_PAIMAI_ORDER_2_CHAIN/3f5f05afc2ae5cb5c6062245d2453ef4/.tmp/0c70c90d40874535b42df5353c8a8bc2 2018-12-13 13:55:11,890 INFO [MemStoreFlusher.3] regionserver.HStore: Added hdfs://pallas/hbase/data/realtime_dw/HB_RT_INDEX_FDM_PAIMAI_PAIMAI_ORDER_2_CHAIN/3f5f05afc2ae5cb5c6062245d2453ef4/d/0c70c90d40874535b42df5353c8a8bc2, entries=11, sequenceid=40209714, filesize=5.9 K 2018-12-13 13:55:11,891 INFO [MemStoreFlusher.3] regionserver.HRegion: Finished memstore flush of ~2.58 KB/2640, currentsize=0 B/0 for region realtime_dw:HB_RT_INDEX_FDM_PAIMAI_PAIMAI_ORDER_2_CHAIN,33333333,1540362902687.3f5f05afc2ae5cb5c6062245d2453ef4. in 16ms, sequenceid=40209714, compaction requested=false 2018-12-13 13:55:14,006 INFO [HT-HBASE-PALLAS-11-7-73-235.hadoop.jd.local,16020,1544603147118_ChoreService_1] regionserver.HRegionServer: HT-HBASE-PALLAS-11-7-73-235.hadoop.jd.local,16020,1544603147118-MemstoreFlusherChore requesting flush for region realtime_dw:HB_RT_INDEX_SERVICE_ID_FDM_AFS_AFS_UNRESOLVED_MESSAGE_CHAIN,5333332e,1541409866093.58218af71c648d8993378d106ae890ae. after a delay of 15422 2018-12-13 13:55:15,124 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: #3, waiting for 1 actions to finish on table: realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN 2018-12-13 13:55:15,124 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Left over 1 task(s) are processed on server(s): [HT-HBASE-PALLAS-11-7-113-76.hadoop.jd.local,16020,1544603147230] 2018-12-13 13:55:15,130 INFO [ReplicationRpcServer.handler=0,queue=0,port=16020] client.AsyncProcess: Regions against which left over task(s) are processed: [realtime_dw:HB_RT_FDM_JD_CHONGZHI_11_CZ_ORDER_N_CHAIN,266616173970c3763836bb5a626fc453_11899118897,1543092690573.3bf4b6ad59e03a6f2d2e5f778138ea29.] 2018-12-13 13:55:21,769 INFO [MemStoreFlusher.3] regionserver.HRegion: Started memstore flush for realtime_dw:HB_RT_INDEX_SERVICE_ID_FDM_AFS_AFS_UNRESOLVED_MESSAGE_CHAIN,5333332e,1541409866093.58218af71c648d8993378d106ae890ae., current region memstore size 96.70 KB, and 1/1 column families' memstores are being flushed.