Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2977

Sharding block map of LogBlockManager

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.12.0
    • Component/s: fs
    • Labels:
      None

      Description

      I found LBM may manage a very large block map when tserver hold large scale data, metric like:

                  {
                      "name": "log_block_manager_blocks_under_management",                                                                                                                                        
                      "value": 7014790 
                  }
      

      LogBlock get/add should hold map's lock first, this is costly and will slow down statup time.

      Todd Lipcon also mentioned in https://issues.apache.org/jira/browse/KUDU-2014.

       

      We can improve it by sharding this big map.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                acelyc111 Yingchun Lai
                Reporter:
                acelyc111 Yingchun Lai
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: