[SPARK-32168] DSv2 SQL overwrite incorrectly uses static plan with hidden partitions

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.1, 3.1.0
• Component/s: SQL
• Labels: correctness
• Target Version/s: 3.0.1
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 03/Jul/20 20:45
• Updated: 24/Nov/20 00:06
• Resolved: 08/Jul/20 23:09

Description

The v2 analyzer rule ResolveInsertInto tries to detect when a static overwrite and a dynamic overwrite would produce the same result and will choose to use static overwrite in that case. It will only use a dynamic overwrite if there is a partition column without a static value and the SQL mode is set to dynamic.

val dynamicPartitionOverwrite = partCols.size > staticPartitions.size &&
          conf.partitionOverwriteMode == PartitionOverwriteMode.DYNAMIC

The problem is that partCols are the names of only partitions that are in the column list (identity partitions) and does not include hidden partitions, like days(ts). As a result, this doesn't detect hidden partitions and use dynamic overwrite. Static overwrite is used instead; when a table has only hidden partitions, the static filter drops all table data.

This is a correctness bug because Spark will overwrite more data than just the set of partitions being written to in dynamic mode. The impact is limited because this rule is only used for SQL queries (not plans from DataFrameWriters) and only affects tables with hidden partitions.

Issue Links

• causes: SPARK-33524 Change `InMemoryTable` not to use Tuple.hashCode for `BucketTransform`
• links to: [Github] Pull Request #28993 (rdblue)