[SPARK-8895] MetricsSystem.removeSource not called in StreamingContext.stop

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.4.0
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: Aniket Bhatnagar
• Created: 08/Jul/15 12:29
• Updated: 09/Jul/15 08:53
• Resolved: 09/Jul/15 08:53

Description

StreamingContext calls env.metricsSystem.registerSource during its construction but does not call env.metricsSystem.removeSource. Therefore, if a user attempts to restart a Streaming job in the same JVM by creating a new instance of StreamingContext with the same application name, it results in exceptions like the following in the log:

[info] o.a.s.m.MetricsSystem -Metrics already registered
java.lang.IllegalArgumentException: A metric named <AppName>.StreamingMetrics.streaming.lastReceivedBatch_processingEndTime already exists
at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:91)~[metrics-core-3.1.0.jar:3.1.0]
at com.codahale.metrics.MetricRegistry.registerAll(MetricRegistry.java:385)~[metrics-core-3.1.0.jar:3.1.0]
at com.codahale.metrics.MetricRegistry.register(MetricRegistry.java:85)~[metrics-core-3.1.0.jar:3.1.0]
at org.apache.spark.metrics.MetricsSystem.registerSource(MetricsSystem.scala:148) ~[spark-core_2.11-1.4.0.jar:1.4.0]
at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:199) [spark-streaming_2.11-1.4.0.jar:1.4.0]
at org.apache.spark.streaming.StreamingContext.<init>(StreamingContext.scala:71) [spark-streaming_2.11-1.4.0.jar:1.4.0]

Issue Links

• duplicates: SPARK-8743 Deregister Codahale metrics for streaming when StreamingContext is closed
• links to: [Github] Pull Request #7317 (aniketbhatnagar)