[SPARK-42112] Add null check before `ContinuousWriteRDD#compute` method close dataWriter

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Yang Jie
• Reporter: Yang Jie
• Created: 19/Jan/23 08:17
• Updated: 20/Jan/23 03:24
• Resolved: 20/Jan/23 03:24

Description

Run 

 

mvn clean test -pl sql/core -Dtest=none -DwildcardSuites=org.apache.spark.sql.streaming.continuous.ContinuousSuite -am

 

NPE exists in the test log

 

- repeatedly restart
...
16:07:39.891 ERROR org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD: Writer for partition 1 is aborting.
16:07:39.891 ERROR org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD: Writer for partition 1 aborted.
16:07:39.892 WARN org.apache.spark.util.Utils: Suppressing exception in finally: null
java.lang.NullPointerException
    at org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD.$anonfun$compute$7(ContinuousWriteRDD.scala:91)
    at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1558)
    at org.apache.spark.sql.execution.streaming.continuous.ContinuousWriteRDD.compute(ContinuousWriteRDD.scala:91)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:328)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:92)
    at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:161)
    at org.apache.spark.scheduler.Task.run(Task.scala:139)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:554)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1502)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:557)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750) 

 

 

The test did not fail because Utils.tryWithSafeFinallyAndFailureCallbacks function suppressed Exception in finally block

 

Issue Links

• links to: [Github] Pull Request #39650 (LuciferYang)