[SEDONA-669] GeoParquet format should handle timestamp_ntz columns properly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.7.0
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Kristin Cowalcijk
• Created: 30/Oct/24 03:57
• Updated: 24/Nov/24 07:58
• Resolved: 24/Nov/24 07:58

Description

A user reported this problem:

In Sedona 1.6.1 (pyspark 3.5), I can write columns of type TimestampNTZType() in parquet format, but if I try writing as a geoparquet it crashes with:

java.lang.RuntimeException: Unsupported data type TimestampNTZType.

The GeoParquet code in sedona was based on the Parquet support code in Spark 3.3, which did not have TimestampNTZType. We can fix this problem by adding TimestampNTZ support to sedona-spark 3.4 and 3.5.

Issue Links

• links to: GitHub Pull Request #1661