[SEDONA-632] Don't use a conventional output committer when writing raster files using df.write.format("raster")

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.1
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Kristin Cowalcijk
• Created: 24/Jul/24 10:05
• Updated: 19/Aug/24 00:31
• Resolved: 19/Aug/24 00:31

Description

Writing large amounts of raster files to distributed file systems or object store is super slow, because the output committer has to move files from temporary locations to their target locations. Users will see all the tasks are completed but the driver is stuck at the committing phase.

We'll add an option useDirectCommitter to the raster format. By default useDirectCommitter is true, and the raster format will use a direct committer that writes raster files to their target locations directly. Users can manually set it to false if they want the original behavior:

df.write.format("raster").option("useDirectCommitter", "false").save("/target/location")

Issue Links

• links to: GitHub Pull Request #1528