[IMPALA-1108] Impala should check the number of opened files/partition during insert

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: Impala 1.4
• Component/s: Frontend
• Labels: ramp-up
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Alan Choi
• Created: 22/Jul/14 19:31
• Updated: 23/Dec/20 03:34
• Resolved: 23/Dec/20 03:34

Description

For insert, when Impala is inserting into a huge number of partition, Impala might be opening too many files. HDFS will return an error, but the error is incomprehensible as "Error(12): Cannot allocate memory".

We can do better to improve the error message. Here are two suggestions:

1. During planning, if there's stats, we know how many partitions are being inserted per Impalad. Based on that, we can determine if we'll be opening too many files. Either return an error or a warning message.

2. During query execution, keep track of the number of files opened for read and write. If we're opening too many files for write, abort the query and returns a proper error message.

Issue Links

• relates to: IMPALA-8125 Limit number of files generated by insert