Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41053 Better Spark UI scalability and Driver stability for large applications
  3. SPARK-42100

Protect null `SQLExecutionUIData#description` in `SQLExecutionUIDataSerializer`

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • SQL
    • None

    Description

      export LIVE_UI_LOCAL_STORE_DIR = /tmp/spark-ui

      mvn clean install -pl sql/core -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest -Dtest=none -DwildcardSuites=org.apache.spark.sql.DynamicPartitionPruningV1SuiteAEOff -am 

       

      no test failed, but some error message:

       

      14:46:44.514 ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener SQLAppStatusListener threw an exception
      java.lang.NullPointerException
          at org.apache.spark.status.protobuf.StoreTypes$SQLExecutionUIData$Builder.setDescription(StoreTypes.java:46500)
          at org.apache.spark.status.protobuf.sql.SQLExecutionUIDataSerializer.serialize(SQLExecutionUIDataSerializer.scala:34)
          at org.apache.spark.status.protobuf.sql.SQLExecutionUIDataSerializer.serialize(SQLExecutionUIDataSerializer.scala:28)
          at org.apache.spark.status.protobuf.KVStoreProtobufSerializer.serialize(KVStoreProtobufSerializer.scala:30)
          at org.apache.spark.util.kvstore.RocksDB.write(RocksDB.java:188)
          at org.apache.spark.status.ElementTrackingStore.write(ElementTrackingStore.scala:123)
          at org.apache.spark.status.ElementTrackingStore.write(ElementTrackingStore.scala:127)
          at org.apache.spark.status.LiveEntity.write(LiveEntity.scala:50)
          at org.apache.spark.sql.execution.ui.SQLAppStatusListener.update(SQLAppStatusListener.scala:456)
          at org.apache.spark.sql.execution.ui.SQLAppStatusListener.onJobStart(SQLAppStatusListener.scala:124)
          at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:37)
          at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
          at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
          at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:37)
          at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:117)
          at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:101)
          at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:105)
          at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:105)
          at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.java:23)
          at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
          at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:100)
          at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:96)
          at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1444)
          at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:96)
      14:46:44.936 ERROR org.apache.spark.scheduler.AsyncEventQueue: Listener SQLAppStatusListener threw an exception 

      Attachments

        Activity

          People

            LuciferYang Yang Jie
            LuciferYang Yang Jie
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: