[IMPALA-1022] Handle cases where in parquet the expected number of rows per metadata is not equal to the actual number of rows in the file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 1.3.1
• Fix Version/s: Impala 1.4
• Labels: impala
• Assignee: Ippokratis Pandis
• Reporter: Ippokratis Pandis
• Created: 30/May/14 23:53
• Updated: 20/Dec/15 00:05
• Resolved: 10/Jun/14 15:01

Description

be/src/exec/hdfs-parquet-scanner.cc:736: rows_read < rows_in_file) {
We should detect the case where file_metadata_.num_row doesn't actually equal the number of rows in the file. If abort_on_error is true, this should fail the query, otherwise we should log something via scan_node_>runtime_state()>LogError().
Such handling did not exist before.
Need to decide whether we will read at most as many rows as the metadata or continue reading until there are no more rows in the file.
We will need to add tests with parquet files whose metadata is not correct.