[SPARK-4294] UnionDStream stream should express the requirements in the same way as TransformedDStream

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Fix Version/s: 1.2.0
• Component/s: DStreams
• Assignee: Tathagata Das
• Reporter: Yadong Qi
• Created: 07/Nov/14 07:14
• Updated: 19/Nov/14 23:53
• Resolved: 19/Nov/14 23:53

Description

In class TransformedDStream:
require(parents.length > 0, "List of DStreams to transform is empty")
require(parents.map(_.ssc).distinct.size == 1, "Some of the DStreams have different contexts")
require(parents.map(_.slideDuration).distinct.size == 1,
"Some of the DStreams have different slide durations")

In class UnionDStream:
if (parents.length == 0)

{ throw new IllegalArgumentException("Empty array of parents") }

if (parents.map(_.ssc).distinct.size > 1)

{ throw new IllegalArgumentException("Array of parents have different StreamingContexts") }

if (parents.map(_.slideDuration).distinct.size > 1)

{ throw new IllegalArgumentException("Array of parents have different slide times") }

The function is the same, but the realization is not. I think they shoule be the same.

Issue Links

• links to: [Github] Pull Request #3151 (watermen)
• links to: [Github] Pull Request #3152 (watermen)