Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.3.4, 2.4.5, 3.0.0
-
None
Description
In `SparkEnv.stop`, the following stop sequence is used.
blockManager.stop() blockManager.master.stop() metricsSystem.stop()
During `metricsSystem.stop`, some sink can invoke `BlockManagerSource` and ends up with NPE.
sinks.foreach(_.stop) registry.removeMatching((_: String, _: Metric) => true)
java.lang.NullPointerException at org.apache.spark.storage.BlockManagerMaster.getStorageStatus(BlockManagerMaster.scala:170) at org.apache.spark.storage.BlockManagerSource$$anonfun$10.apply(BlockManagerSource.scala:63) at org.apache.spark.storage.BlockManagerSource$$anonfun$10.apply(BlockManagerSource.scala:63) at org.apache.spark.storage.BlockManagerSource$$anon$1.getValue(BlockManagerSource.scala:31) at org.apache.spark.storage.BlockManagerSource$$anon$1.getValue(BlockManagerSource.scala:30)
Since SparkContext registers and forgets `BlockManagerSource` without deregistering, we had better avoid NPE inside `BlockManagerMaster`.
_env.metricsSystem.registerSource(new BlockManagerSource(_env.blockManager))
Attachments
Issue Links
- links to