Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5378

Disk IO manager needs to understand ADLS

        Details

        • Type: Bug
        • Status: Resolved
        • Priority: Critical
        • Resolution: Fixed
        • Affects Version/s: Impala 2.9.0
        • Fix Version/s: Impala 2.9.0
        • Component/s: Backend
        • Labels:
          None

          Description

          The Disk IO Manager has customized support for S3 and remote HDFS that allows for these to use a separate queue and have a customized number of IO threads. Currently, ADLS does not have this support.

          Based on the code in DiskIoMgr::Init and DiskIoMgr::AssignQueue, IOs for ADLS would currently be put in local disk queues. Since local disks are considered rotational unless we can confirm otherwise by looking at the /sys filesystem, this means that THREADS_PER_ROTATIONAL_DISK=1 will apply as the thread count.

          This reduced thread count may hurt ADLS performance.

            Activity

            Hide
            Permalink
            sailesh Sailesh Mukil added a comment -

            Commit in:
            https://github.com/apache/incubator-impala/commit/9c4f75a6a877f2b5e56d24f1e73fcf72d74f9129

            Show
            sailesh Sailesh Mukil added a comment - Commit in: https://github.com/apache/incubator-impala/commit/9c4f75a6a877f2b5e56d24f1e73fcf72d74f9129

              People

              • Assignee:
                sailesh Sailesh Mukil
                Reporter:
                joemcdonnell Joe McDonnell
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development