Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31584

NullPointerException when parsing event log with InMemoryStore

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 3.0.1
    • Fix Version/s: 3.0.1, 3.1.0
    • Component/s: Web UI
    • Labels:


      I compiled with the current branch-3.0 source and tested it in mac os. A java.lang.NullPointerException will be thrown when below conditions are met: 

      1. Using InMemoryStore as kvstore when parsing the event log file (e.g., when spark.history.store.path is unset). 
      2. At least one stage in this event log has task number greater than spark.ui.retainedTasks (by default is 100000). In this case, kvstore needs to delete extra task records.
      3. The job has more than one stage, so parentToChildrenMap in InMemoryStore.java will have more than one key.

      The java.lang.NullPointerException is thrown in InMemoryStore.java :296. In the method deleteParentIndex().

          private void deleteParentIndex(Object key) {
            if (hasNaturalParentIndex) {
              for (NaturalKeys v : parentToChildrenMap.values()) {
                if (v.remove(asKey(key))) {
                  // `v` can be empty after removing the natural key and we can remove it from
                  // `parentToChildrenMap`. However, `parentToChildrenMap` is a ConcurrentMap and such
                  // checking and deleting can be slow.
                  // This method is to delete one object with certain key, let's make it simple here.

      In “if (v.remove(asKey(key)))”, if the key is not contained in v,  "v.remove(asKey(key))" will return null, and java will throw a NullPointerException when executing "if (null)".

      An exception stack trace is attached.

      This issue can be fixed by updating if statement to

      if (v.remove(asKey(key)) != null)




            • Assignee:
              Baohe Zhang Baohe Zhang
              Baohe Zhang Baohe Zhang


              • Created:

                Issue deployment