[IMPALA-4835] HDFS scans should operate with a constrained number of I/O buffers

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 2.13.0, Impala 3.1.0
• Component/s: Backend
• Labels: resource-management
• Epic Link: Operator memory constraints
• Target Version: Impala 3.1.0
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 27/Jan/17 22:55
• Updated: 16/Aug/18 21:34
• Resolved: 30/Apr/18 16:20

Description

The HDFS scan nodes should be able to operate with a fixed number of I/O buffers in most cases (excluding very large rows). We should modify them to claim a reservation upfront and use this for all disk I/O.

This probably also requires switching DiskIoMgr to allocate memory from BufferPool.

Issue Links

• blocks: IMPALA-6460 More flexible memory-based admission control policies
• breaks: IMPALA-6587 Crash in DiskMgr::AllocateBuffersForRange
• breaks: IMPALA-6592 Fix test gap - no test of handling for invalid/unsupported Parquet codec
• breaks: IMPALA-6594 Various tests failing (some flaky) with memory limit exceeded
• breaks: IMPALA-6598 Investigate memory requirement regression from scanner buffer pool change
• breaks: IMPALA-6588 test_compute_stats_tablesample failing with "Cancelled"
• breaks: IMPALA-6585 test_low_mem_limit_q21 flaky under ASAN
• depends upon: IMPALA-6587 Crash in DiskMgr::AllocateBuffersForRange
• depends upon: IMPALA-6592 Fix test gap - no test of handling for invalid/unsupported Parquet codec
• depends upon: IMPALA-6594 Various tests failing (some flaky) with memory limit exceeded
• depends upon: IMPALA-6598 Investigate memory requirement regression from scanner buffer pool change
• depends upon: IMPALA-5389 Clarify lifetime of DiskIoMgr::BufferDescriptor objects
• is blocked by: IMPALA-6564 Queries randomly fail with "CANCELLED" due to a race with IssueInitialRanges()
• is depended upon by: IMPALA-6673 Be smarter about I/O patterns for Parquet scan ranges
• is depended upon by: IMPALA-6875 Reservations for ORC scanner
• is depended upon by: IMPALA-6555 Clean up relationship between DiskIoMgr::min_buffer_size_ and BufferPool::min_buffer_len_
• is duplicated by: IMPALA-3209 Optimize scanner memory usage and make scan nodes adhere to a memory constraint

Sub-Tasks

• Consider always copying-out Disk I/O buffers instead of attaching to RowBatches: Resolved
• Consider limiting I/O buffer queue size to 2 buffers: Resolved
• Simplify ScannerContext buffer management to only use one I/O buffer at a time.: Resolved
• Crash in DiskMgr::AllocateBuffersForRange: Resolved
• Various tests failing (some flaky) with memory limit exceeded: Resolved
• Investigate memory requirement regression from scanner buffer pool change: Resolved
• Fix test gap - no test of handling for invalid/unsupported Parquet codec: Resolved
• Better estimate of per-column compressed data size for low-NDV columns.: Resolved
• Don't claim ideal reservation in scanner until actually processing scan ranges: Resolved
• Consider reserving max 2 I/O buffers per Parquet column when there are multiple columns: Resolved