[KUDU-3253] Log block manager may consume a huge amount of memory

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: fs
• Assignee: Unassigned
• Reporter: Li Zhiming
• Created: 01/Mar/21 08:20
• Updated: 01/Mar/21 08:26

Description

LBM kept low live ratio containers' metadata for compation and punching hole in memory. The amount may grow too large，makes the time wasted on handling page fault.

 
I0223 21:30:07.741448 12842 log_block_manager.cc:2746] Opened 6200 log block containers
I0223 21:47:32.162726 12845 log_block_manager.cc:2746] Opened 6400 log block containers
I0224 08:29:40.393473 54898 log_block_manager.cc:2975] Compacted 2333 metadata files (279182030063 metadata bytes)
I0224 10:34:03.299155 73062 log_block_manager.cc:2975] Compacted 2337 metadata files (280053530024 metadata bytes)
I0224 10:47:21.935983 12825 fs_manager.cc:433] Time spent opening block manager: real 62879.018s user 1111.182s sys 1195.374s