[SPARK-31584] NullPointerException when parsing event log with InMemoryStore

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.1
• Fix Version/s: 3.0.1, 3.1.0
• Component/s: Web UI
• Target Version/s: 3.0.0, 3.0.1, 3.1.0
• Assignee: Baohe Zhang
• Reporter: Baohe Zhang
• Created: 28/Apr/20 00:16
• Updated: 23/Jun/20 05:16
• Resolved: 29/Apr/20 00:30

Description

I compiled with the current branch-3.0 source and tested it in mac os. A java.lang.NullPointerException will be thrown when below conditions are met: 

1. Using InMemoryStore as kvstore when parsing the event log file (e.g., when spark.history.store.path is unset). 
2. At least one stage in this event log has task number greater than spark.ui.retainedTasks (by default is 100000). In this case, kvstore needs to delete extra task records.
3. The job has more than one stage, so parentToChildrenMap in InMemoryStore.java will have more than one key.

The java.lang.NullPointerException is thrown in InMemoryStore.java :296. In the method deleteParentIndex().

    private void deleteParentIndex(Object key) {
      if (hasNaturalParentIndex) {
        for (NaturalKeys v : parentToChildrenMap.values()) {
          if (v.remove(asKey(key))) {
            // `v` can be empty after removing the natural key and we can remove it from
            // `parentToChildrenMap`. However, `parentToChildrenMap` is a ConcurrentMap and such
            // checking and deleting can be slow.
            // This method is to delete one object with certain key, let's make it simple here.
            break;
          }
        }
      }
    }

In “if (v.remove(asKey(key)))”, if the key is not contained in v,  "v.remove(asKey(key))" will return null, and java will throw a NullPointerException when executing "if (null)".

An exception stack trace is attached.

This issue can be fixed by updating if statement to

if (v.remove(asKey(key)) != null)

Issue Links

• links to: Pull Request #28378