[SPARK-23819] InMemoryTableScanExec prunes orderable complex types due to out of date ColumnStats

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.3.0
• Component/s: SQL
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Patrick Woody
• Created: 29/Mar/18 10:46
• Updated: 25/May/21 01:50
• Resolved: 25/May/21 01:39

Description

The data types that can be compared via BinaryComparison was expanded in SPARK-21110 now include Arrays/Structs/etc, but ColumnStats would still have hard coded upper/lower bounds for these types.

InMemoryTableScanExec used to be safe against these comparisons because the predicate would fail type checking. Now that it passes, the statistics unintentionally allow pruning of the partition, causing correctness issues.

Issue Links

• links to: [Github] Pull Request #20935 (pwoody)
• links to: GitHub Pull Request #20935