[KUDU-2977] Sharding block map of LogBlockManager

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.12.0
• Component/s: fs
• Assignee: Yingchun Lai
• Reporter: Yingchun Lai
• Created: 17/Oct/19 09:05
• Updated: 19/Nov/19 09:19
• Resolved: 13/Nov/19 07:17

Description

I found LBM may manage a very large block map when tserver hold large scale data, metric like:

            {
                "name": "log_block_manager_blocks_under_management",                                                                                                                                        
                "value": 7014790 
            }

LogBlock get/add should hold map's lock first, this is costly and will slow down statup time.

tlipcon also mentioned in https://issues.apache.org/jira/browse/KUDU-2014.

 

We can improve it by sharding this big map.

Issue Links

• Blocked: KUDU-2014 Explore additional approaches to improve LBM startup time