[SPARK-36137] HiveShim always fallback to getAllPartitionsOf regardless of whether directSQL is enabled in remote HMS

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.3.0
• Component/s: SQL
• Assignee: Chao Sun
• Reporter: Chao Sun
• Created: 14/Jul/21 20:48
• Updated: 09/Dec/21 04:38
• Resolved: 02/Aug/21 23:49

Description

At the moment getPartitionsByFilter in Hive shim only fallback to use getAllPartitionsOf when hive.metastore.try.direct.sql is enabled in the remote HMS. However, in certain cases the remote HMS will fallback to use ORM (which only support string type for partition columns) to query the underlying RDBMS even if this config is set to true, and Spark will not be able to recover from the error and will just fail the query.

For instance, we encountered this bug HIVE-21497 in HMS running Hive 3.1.2, and Spark was not able to pushdown filter for date column.

Issue Links

• is related to: SPARK-17992 HiveClient.getPartitionsByFilter throws an exception for some unsupported filters when hive.metastore.try.direct.sql=false
• is related to: SPARK-33477 Hive partition pruning support date type
• links to: [Github] Pull Request #33382 (sunchao)
• links to: [Github] Pull Request #33382 (sunchao)
• links to: [Github] Pull Request #34843 (ulysses-you)