Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5378

Disk IO manager needs to understand ADLS

          Details

          • Type: Bug
          • Status: Resolved
          • Priority: Critical
          • Resolution: Fixed
          • Affects Version/s: Impala 2.9.0
          • Fix Version/s: Impala 2.9.0
          • Component/s: Backend
          • Labels:
            None

            Description

            The Disk IO Manager has customized support for S3 and remote HDFS that allows for these to use a separate queue and have a customized number of IO threads. Currently, ADLS does not have this support.

            Based on the code in DiskIoMgr::Init and DiskIoMgr::AssignQueue, IOs for ADLS would currently be put in local disk queues. Since local disks are considered rotational unless we can confirm otherwise by looking at the /sys filesystem, this means that THREADS_PER_ROTATIONAL_DISK=1 will apply as the thread count.

            This reduced thread count may hurt ADLS performance.

              Attachments

                Activity

                  People

                  • Assignee:
                    sailesh Sailesh Mukil
                    Reporter:
                    joemcdonnell Joe McDonnell
                  • Votes:
                    0 Vote for this issue
                    Watchers:
                    2 Start watching this issue

                    Dates

                    • Created:
                      Updated:
                      Resolved: