[SEDONA-470] Cannot distinguish between missing or null crs from the result of geoparquet.metadata

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5.2
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Kristin Cowalcijk
• Created: 20/Jan/24 04:02
• Updated: 28/Apr/24 08:07
• Resolved: 28/Apr/24 08:07

Description

geoparquet.metadata always gives crs = NULL no matter if the crs metadata field is missing or is explicitly set to null. The result of running

spark.read.format("geoparquet.metadata").load("path/to/geoparquet").selectExpr("columns").show(10, False)

is always

+------------------------------------------------------------+
|columns                                                     |
+------------------------------------------------------------+
|{geom -> {WKB, [Polygon], [0.0, 0.0, 1000.0, 1000.0], NULL}}|
+------------------------------------------------------------+

No matter if the geoparquet files has missing crs or having crs explicitly set to null.

To better distinguish these cases, geoparquet.metadata should return an empty string for missing crs fields, and returns "null" for crs fields explicitly set to null.

For example, for geoparquet with missing crs metadata, the output should be:

+--------------------------------------------------------+
|columns                                                 |
+--------------------------------------------------------+
|{geom -> {WKB, [Polygon], [0.0, 0.0, 1000.0, 1000.0], }}|
+--------------------------------------------------------+

For geoparquet with crs explicitly set to null, the output should be:

+------------------------------------------------------------+
|columns                                                     |
+------------------------------------------------------------+
|{geom -> {WKB, [Polygon], [0.0, 0.0, 1000.0, 1000.0], null}}|
+------------------------------------------------------------+

For geoparquet with CRS, the output should be:

+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|columns                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|{geometry -> {WKB, [Polygon, MultiPolygon], [-180.0, -90.0, 180.0, 83.6451], {"$schema":"https://proj.org/schemas/v0.5/projjson.schema.json","type":"GeographicCRS","name":"WGS 84 (CRS84)","datumEnsemble":{"name":"World Geodetic System 1984 ensemble","members":[{"name":"World Geodetic System 1984 (Transit)","id":{"authority":"EPSG","code":1166}},{"name":"World Geodetic System 1984 (G730)","id":{"authority":"EPSG","code":1152}},{"name":"World Geodetic System 1984 (G873)","id":{"authority":"EPSG","code":1153}},{"name":"World Geodetic System 1984 (G1150)","id":{"authority":"EPSG","code":1154}},{"name":"World Geodetic System 1984 (G1674)","id":{"authority":"EPSG","code":1155}},{"name":"World Geodetic System 1984 (G1762)","id":{"authority":"EPSG","code":1156}},{"name":"World Geodetic System 1984 (G2139)","id":{"authority":"EPSG","code":1309}}],"ellipsoid":{"name":"WGS 84","semiMajorAxis":6378137,"inverseFlattening":298.257223563},"accuracy":"2.0","id":{"authority":"EPSG","code":6326}},"coordinateSystem":{"subtype":"ellipsoidal","axis":[{"name":"Geodetic longitude","abbreviation":"Lon","direction":"east","unit":"degree"},{"name":"Geodetic latitude","abbreviation":"Lat","direction":"north","unit":"degree"}]},"scope":"Not known.","area":"World.","bbox":{"southLatitude":-90,"westLongitude":-180,"northLatitude":90,"eastLongitude":180},"id":{"authority":"OGC","code":"CRS84"}}}}|
+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Issue Links

• links to: GitHub Pull Request #1211