Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1489

Use WAL directory for tablet metadata files

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.9.0
    • 1.7.0
    • consensus, fs, tserver
    • None

    Description

      Today a tserver will place tablet metadata files (i.e. superblock and cmeta files) in the first configured data directory. I don't remember why we decided to do this (commit 691f97d introduced it), but upon reconsideration the WAL directory seems like a much better choice, because if the machine has different kinds of I/O devices, the WAL directory's device is typically the fastest.

      Mostafa has been testing Impala and Kudu on a cluster with many thousands of tablets. His cluster contains storage-dense machines, each configured with 14 spinning disks and one flash device. Naturally, the WAL directory sits on that flash device and the data directories are on the spinning disks. With thousands of tablet metadata files on the first spinning disk, nearly every tablet in the tserver is bottlenecked on that device due to the sheer amount of I/O needed to maintain the running state of the tablet, specifically rewriting cmeta files on various Raft events (votes, term advancement, etc.).

      Many thousands of tablets is not really a good scale for Kudu right now, but moving the tablet metadata files to a faster device should at least help with the above.

      Attachments

        Issue Links

          Activity

            People

              awong Andrew Wong
              adar Adar Dembo
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: