[IMPALA-525] Scanners use too much memory when reading past scan range

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 1.1
• Fix Version/s: Impala 1.2.1
• Assignee: Skye Wanderman-Milne
• Reporter: Skye Wanderman-Milne
• Created: 12/Aug/13 21:57
• Updated: 20/Dec/15 00:05
• Resolved: 19/Nov/13 01:09

Description

The default read-ahead size for scanners is very small, so they end up issuing many small reads past the end of the scan range, each of which uses an 8MB IO buffer. This eventually can cause the process limit to be exceeded and cause queries to fail with "Memory limit exceeded".

This problem is especially bad for uncompressed RCFiles, which have large data blocks.