[SEDONA-495] Raster data source uses shared FileSystem connections which lead to race condition

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0, 1.4.1, 1.5.0, 1.5.1
• Fix Version/s: 1.5.2
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Jia Yu
• Created: 15/Feb/24 01:02
• Updated: 28/Apr/24 08:12
• Resolved: 28/Apr/24 08:12

Description

The raster data source's OutputWriter uses `new Path(savePath).getFileSystem(context.getConfiguration)` to get a Hadoop FileSystem instance and a OutputWriter instance is initiated per task. This function will return a shared connection among all tasks on an executor.

 

https://github.com/apache/sedona/blob/master/spark/common/src/main/scala/org/apache/spark/sql/sedona_sql/io/raster/RasterFileFormat.scala#L85

 

It is common that a multi-core executor gets multiple concurrent tasks (one task per core). In the current implementation, if one task is completed, the connection is closed and all other tasks are having IO exception.

 

The best practice is to use `FileSystem.newInstance` for each task.

 

Issue Links

• links to: GitHub Pull Request #1236