[SEDONA-494] Raster data source cannot write to HDFS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0, 1.4.1, 1.5.0, 1.5.1
• Fix Version/s: 1.5.2
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Jia Yu
• Created: 09/Feb/24 22:10
• Updated: 28/Apr/24 08:12
• Resolved: 28/Apr/24 08:12

Description

Reproduce

 

When run the following code

var df = spark.read.format("binaryFile").load("/user/spark/raster/input.tif")
df.write.format("raster").mode(org.apache.spark.sql.SaveMode.Overwrite).save("output")

 

Just a _SUCCESS file found in the path.

I can find tiff file created in HDFS audit.log , but there's not 'rename' cmd .
I can find "SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_xxx
BasicWriteTaskStatsTracker: Expected 1 files, but only saw 0. This could be due to the output format not writing empty files, or files being not immediately visible in the filesystem." in executor log.

 

Solution:

in "org.apache.spark.sql.sedona_sql.io.raster.RasterFileFormat.scala"
val out = hfs.create(new Path(Paths.get(savePath, new Path(rasterFilePath).getName).toString))
=>
val out = hfs.create(new Path(savePath, new Path(rasterFilePath).getName))
will solve the problem

Paths.get should not be used in FileSystem implements

Issue Links

• links to: GitHub Pull Request #1235