[SPARK-43188] Cannot write to Azure Datalake Gen2 (abfs/abfss) after Spark 3.1.2

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Workaround
• Affects Version/s: 3.3.2, 3.4.0
• Component/s: PySpark, Spark Core
• Assignee: Unassigned
• Reporter: Nicolas PHUNG
• Created: 19/Apr/23 12:20
• Updated: 26/May/23 15:52
• Resolved: 26/May/23 15:52

Description

Hello,

I have an issue with Spark 3.3.2 & Spark 3.4.0 to write into Azure Data Lake Storage Gen2 (abfs/abfss scheme). I've got the following errors:

warn 13:12:47.554: StdErr from Kernel Process 23/04/19 13:12:47 ERROR FileFormatWriter: Aborting job 6a75949c-1473-4445-b8ab-d125be3f0f21.org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 0.0 failed 1 times, most recent failure: Lost task 1.0 in stage 0.0 (TID 1) (myhost executor driver): org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for datablock-0001-    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:462)    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:165)    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)    at org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory.createTmpFileForWrite(DataBlocks.java:980)    at org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory.create(DataBlocks.java:960)    at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.createBlockIfNeeded(AbfsOutputStream.java:262)    at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.<init>(AbfsOutputStream.java:173)    at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.createFile(AzureBlobFileSystemStore.java:580)    at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.create(AzureBlobFileSystem.java:301)    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)    at org.apache.parquet.hadoop.util.HadoopOutputFile.create(HadoopOutputFile.java:74)    at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:347)    at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:314)    at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:480)    at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:420)    at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:409)    at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:36)    at org.apache.spark.sql.execution.datasources.parquet.ParquetUtils$$anon$1.newInstance(ParquetUtils.scala:490)    at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:161)    at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:146)    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:389)    at org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:100)    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:888)    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:888)    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)    at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)    at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)    at org.apache.spark.scheduler.Task.run(Task.scala:139)    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)    at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:    at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2785)    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2721)    at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2720)    at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)    at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2720)    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1206)    at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1206)    at scala.Option.foreach(Option.scala:407)    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1206)    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2984)    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2923)    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2912)    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:971)    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2263)    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeWrite$4(FileFormatWriter.scala:307)    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.writeAndCommit(FileFormatWriter.scala:271)    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeWrite(FileFormatWriter.scala:304)    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:190)    at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:190)    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult$lzycompute(commands.scala:113)    at org.apache.spark.sql.execution.command.DataWritingCommandExec.sideEffectResult(commands.scala:111)    at org.apache.spark.sql.execution.command.DataWritingCommandExec.executeCollect(commands.scala:125)    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98)    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:118)    at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195)    at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103)    at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:827)    at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65)    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:98)    at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.applyOrElse(QueryExecution.scala:94)    at org.apache.spark.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:512)    at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:104)    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDownWithPruning(TreeNode.scala:512)    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.org$apache$spark$sql$catalyst$plans$logical$AnalysisHelper$$super$transformDownWithPruning(LogicalPlan.scala:31)    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning(AnalysisHelper.scala:267)    at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.transformDownWithPruning$(AnalysisHelper.scala:263)    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)    at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.transformDownWithPruning(LogicalPlan.scala:31)    at org.apache.spark.sql.catalyst.trees.TreeNode.transformDown(TreeNode.scala:488)    at org.apache.spark.sql.execution.QueryExecution.eagerlyExecuteCommands(QueryExecution.scala:94)    at org.apache.spark.sql.execution.QueryExecution.commandExecuted$lzycompute(QueryExecution.scala:81)    at org.apache.spark.sql.execution.QueryExecution.commandExecuted(QueryExecution.scala:79)    at org.apache.spark.sql.execution.QueryExecution.assertCommandExecuted(QueryExecution.scala:133)    at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:856)    at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:387)    at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:360)    at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239)    at org.apache.spark.sql.DataFrameWriter.parquet(DataFrameWriter.scala:789)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)    at java.lang.reflect.Method.invoke(Method.java:498)    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)    at py4j.Gateway.invoke(Gateway.java:282)    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)    at py4j.commands.CallCommand.execute(CallCommand.java:79)    at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182)    at py4j.ClientServerConnection.run(ClientServerConnection.java:106)    at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find any valid local directory for datablock-0001-    at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:462)    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:165)    at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:146)    at org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory.createTmpFileForWrite(DataBlocks.java:980)    at org.apache.hadoop.fs.store.DataBlocks$DiskBlockFactory.create(DataBlocks.java:960)    at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.createBlockIfNeeded(AbfsOutputStream.java:262)    at org.apache.hadoop.fs.azurebfs.services.AbfsOutputStream.<init>(AbfsOutputStream.java:173)    at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.createFile(AzureBlobFileSystemStore.java:580)    at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.create(AzureBlobFileSystem.java:301)    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1195)    at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)    at org.apache.parquet.hadoop.util.HadoopOutputFile.create(HadoopOutputFile.java:74)    at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:347)    at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:314)    at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:480)    at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:420)    at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:409)    at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetOutputWriter.scala:36)    at org.apache.spark.sql.execution.datasources.parquet.ParquetUtils$$anon$1.newInstance(ParquetUtils.scala:490)    at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.newOutputWriter(FileFormatDataWriter.scala:161)    at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.<init>(FileFormatDataWriter.scala:146)    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:389)    at org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:100)    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:888)    at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:888)    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)    at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)    at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)    at org.apache.spark.scheduler.Task.run(Task.scala:139)    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1529)    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)    ... 1 more 

Before that, I was able to write into Azure Data Lake Storage with Spark 3.1.2 with hadoop-azure 3.2.1 without encountering this error.

Here's what I have tried but with no success:

• Spark 3.3.2 with hadoop-azure 3.3.2
• Spark 3.3.2 with hadoop-azure 3.3.5
• Spark 3.4.0 with hadoop-azure 3.3.4
• Spark 3.4.0 with hadoop-azure 3.3.5

Regards,

 

Issue Links

• is blocked by: HADOOP-18707 Cannot write to Azure Datalake Gen2 (abfs/abfss) after Spark 3.1.2
• is caused by: HADOOP-17195 Intermittent OutOfMemory error while performing hdfs CopyFromLocal to abfs