[HBASE-21575] memstore above high watermark message is logged too much

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 1.5.0, 2.2.0
• Component/s: logging, regionserver
• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 10/Dec/18 23:28
• Updated: 23/Jan/19 15:45
• Resolved: 13/Dec/18 20:54

Description

100s of Mb of logs like this, in a tight loop:

2018-12-08 10:27:00,462 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3646ms
2018-12-08 10:27:00,463 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3647ms
2018-12-08 10:27:00,463 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3647ms
2018-12-08 10:27:00,464 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3648ms
2018-12-08 10:27:00,464 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3648ms
2018-12-08 10:27:00,465 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3649ms
2018-12-08 10:27:00,465 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3649ms
2018-12-08 10:27:00,466 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3650ms
2018-12-08 10:27:00,466 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3650ms
2018-12-08 10:27:00,467 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3651ms
2018-12-08 10:27:00,469 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3653ms
2018-12-08 10:27:00,470 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3654ms
2018-12-08 10:27:00,470 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3654ms
2018-12-08 10:27:00,471 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3655ms
2018-12-08 10:27:00,471 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3655ms
2018-12-08 10:27:00,472 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3656ms
2018-12-08 10:27:00,472 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3656ms
2018-12-08 10:27:00,473 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3657ms
2018-12-08 10:27:00,474 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3658ms
2018-12-08 10:27:00,475 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3659ms
2018-12-08 10:27:00,476 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3660ms
2018-12-08 10:27:00,476 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3660ms
2018-12-08 10:27:00,477 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3661ms
2018-12-08 10:27:00,477 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3661ms
2018-12-08 10:27:00,478 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3662ms
2018-12-08 10:27:00,479 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3663ms
2018-12-08 10:27:00,479 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3663ms
2018-12-08 10:27:00,480 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3664ms
2018-12-08 10:27:00,480 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3664ms
2018-12-08 10:27:00,481 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3665ms
2018-12-08 10:27:00,482 WARN  [RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=17020] regionserver.MemStoreFlusher: Memstore is above high water mark and block 3666ms