[IMPALA-5378] Disk IO manager needs to understand ADLS

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 2.9.0
• Component/s: Backend
• Target Version: Impala 2.9.0
• Epic Color: ghx-label-8
• Assignee: Sailesh Mukil
• Reporter: Joe McDonnell
• Created: 26/May/17 17:21
• Updated: 02/Jun/17 19:54
• Resolved: 02/Jun/17 19:54

Description

The Disk IO Manager has customized support for S3 and remote HDFS that allows for these to use a separate queue and have a customized number of IO threads. Currently, ADLS does not have this support.

Based on the code in DiskIoMgr::Init and DiskIoMgr::AssignQueue, IOs for ADLS would currently be put in local disk queues. Since local disks are considered rotational unless we can confirm otherwise by looking at the /sys filesystem, this means that THREADS_PER_ROTATIONAL_DISK=1 will apply as the thread count.

This reduced thread count may hurt ADLS performance.