Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39847

Race condition related to interruption of task threads while they are in RocksDBLoader.loadLibrary()

    XMLWordPrintableJSON

Details

    Description

      One of our workloads experienced a rare failure in `RocksDBLoader`

      Caused by: java.lang.IllegalThreadStateException
      	at java.lang.Thread.start(Thread.java:708)
      	at org.apache.spark.sql.execution.streaming.state.RocksDBLoader$.loadLibrary(RocksDBLoader.scala:51) 

      After investigation, we determined that was due to task cancellation: if the task which starts the RocksDB library loading is interrupted, another thread may begin a load and crash with the thread state exception.

      Skimming through the code in RocksDBLoader, I spot a potential race condition:

      • Although the native JNI call is uninterruptible, the thread which calls loadLibrary is still interruptible. Let’s call that thread the “task thread”.
      • Say we have two tasks, A and B, which both want to load the JNI library.
      • Say that Task A wins the race to perform the load and enters the synchronized block in loadLibrary(), spawns a child thread to perform the actual loading, then blocks in the loadLibraryThread.join() call.
      • If Task A is interrupted, an InterruptedException will be thrown and it will exit the loadLibrary synchronized block.
      • At this point, Task B enters the synchronized block and sees that exception == null because the loading thread is still running, so it calls loadLibraryThread.start() and hits the thread state error.

      One way to fix this is to add

       if (loadLibraryThread.getState == Thread.State.NEW) {
              loadLibraryThread.start()
            }

      to ensure that only one thread starts the loadLibraryThread. If the original starter thread is interrupted then a new thread will encounter this block, skip the start(), proceed to the join() and block on the original load thread.

      I will submit a PR with this fix.

      Attachments

        Activity

          People

            joshrosen Josh Rosen
            joshrosen Josh Rosen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: