Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5378

Disk IO manager needs to understand ADLS

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
      None

      Description

      The Disk IO Manager has customized support for S3 and remote HDFS that allows for these to use a separate queue and have a customized number of IO threads. Currently, ADLS does not have this support.

      Based on the code in DiskIoMgr::Init and DiskIoMgr::AssignQueue, IOs for ADLS would currently be put in local disk queues. Since local disks are considered rotational unless we can confirm otherwise by looking at the /sys filesystem, this means that THREADS_PER_ROTATIONAL_DISK=1 will apply as the thread count.

      This reduced thread count may hurt ADLS performance.

        Activity

        Show
        sailesh Sailesh Mukil added a comment - Commit in: https://github.com/apache/incubator-impala/commit/9c4f75a6a877f2b5e56d24f1e73fcf72d74f9129

          People

          • Assignee:
            sailesh Sailesh Mukil
            Reporter:
            joemcdonnell Joe McDonnell
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development