Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5169

Parallelise read I/O of BufferPool::Pin()

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
      None

      Description

      Currently read I/O in BufferPool is synchronous. In some cases this can lead to poor resource utilisation and I/O throughput, because:

      • We don't dispatch parallel reads to multiple scratch disks or high-throughput SSDs
      • Issuing reads of contiguous scratch ranges at the same time improves the odds that the second read can be served without a disk seek or by the disks internal cache.
      • Expose a batched Pin() interface that can pin multiple buffers at the same time
      • Expose an asynchronous Pin() interface that can start the read, and allow the client to wait for it.

      The first alternative is probably simplest.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: