[IMPALA-3014] Hdfs scan node should wait for all filters, not just partition filters

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.5.0
• Fix Version/s: Impala 2.5.0
• Component/s: Distributed Exec
• Labels: runtime-filters
• Target Version: Impala 2.5.0
• Assignee: Henry Robinson
• Reporter: Henry Robinson
• Created: 17/Feb/16 01:24
• Updated: 19/Feb/16 19:50
• Resolved: 19/Feb/16 19:50

Description

HdfsScanNode::WaitForPartitionFilters() shouldn't only wait for partition filters, as there is often a lot of value in doing row-based filtering as well.

However, we should also avoid waiting for filters that we can't apply (e.g. if row filtering is turned off, or if the file format is not parquet). The best thing to do is probably to wait separately, in the parquet column reader, since that gives row filters the maximum time to arrive and doesn't affect waiting for partition filters earlier in scanner's pipeline.