[SPARK-43357] Spark AWS Glue date partition push down broken

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.1.3, 3.2.1, 3.3.0, 3.2.2, 3.3.1, 3.2.3, 3.2.4, 3.3.2
• Fix Version/s: 3.5.0
• Component/s: SQL
• Assignee: Stijn De Haes
• Reporter: Stijn De Haes
• Created: 03/May/23 14:03
• Updated: 27/Dec/23 18:32
• Resolved: 09/May/23 13:06

Description

When using the following project: https://github.com/awslabs/aws-glue-data-catalog-client-for-apache-hive-metastore
To have glue supported as as a hive metastore for spark there is an issue when reading a date-partitioned data set. Writing is fine.
You get the following error: 

org.apache.hadoop.hive.metastore.api.InvalidObjectException: Unsupported expression '2023 - 05 - 03' (Service: AWSGlue; Status Code: 400; Error Code: InvalidInputException; Request ID: beed68c6-b228-442e-8783-52c25b9d2243; Proxy: null)

 

A fix for this is making sure the date passed to glue is quoted

Issue Links

• links to: [Github] Pull Request #41035 (stijndehaes)