[IMPALA-2209] More user-friendly behavior for mismatched table format and file format

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: Impala 2.2.4
• Component/s: Backend
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: John Russell
• Created: 14/Aug/15 19:14
• Updated: 10/Jan/17 20:06
• Resolved: 27/Sep/16 22:26

Description

Currently, if you put a Parquet file into a text table, queries can experience long pauses and throw lots of conversion errors. SELECT COUNT will even return a number, even though wildly inaccurate.

Could the scanner recognize a magic number in the file header for Parquet, Avro, SequenceFile, and/or RCFile and either fail the query or skip the file if it was the wrong format relative to the table or partition?

Now that I think about it, this might involve a little complexity if node X read the first block of the file containing the magic number, while node Y read the second block. Would the coordinator node have some way to detect the mismatch and communicate to all the other nodes to skip such-and-such a file or block? Perhaps this technique would be easiest to implement for Parquet files, with only a single block to consider.

Issue Links

• is related to: IMPALA-4753 Table created like parquet file shows wrong row count