[SPARK-6331] New Spark Master URL is not picked up when streaming context is started from checkpoint

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.1.1, 1.2.1, 1.3.0
• Component/s: DStreams
• Target Version/s: 1.3.1, 1.4.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 13/Mar/15 21:56
• Updated: 17/Mar/15 12:32
• Resolved: 17/Mar/15 12:32

Description

When the SparkConf is reconstructed based on the checkpointed configuration, it recovers the old master URL. This okay if the cluster on which the streaming application is relaunched is the same cluster as it was running before. But if that cluster changes, there is no way to inject the new master URL of the new cluster. As a result, the restarted app tries to connect to the non-existent old cluster and fails.

The solution is to check whether a master URL is set in the System properties (by Spark submit) before recreating the SparkConf. If a new master url is set in the properties, then use it as that is obviously the most relevant one. Otherwise load the old one (to maintain existing behavior).

Issue Links

• links to: [Github] Pull Request #5024 (tdas)