[SPARK-21206] the window slice of Dstream is wrong

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.1.0
• Component/s: DStreams
• Assignee: Unassigned
• Reporter: Fei Shao
• Created: 25/Jun/17 09:42
• Updated: 03/Jul/17 12:02
• Resolved: 02/Jul/17 07:59

Description

the code is :

val conf = new SparkConf().setAppName("testDstream").setMaster("local[4]")
val ssc = new StreamingContext(conf, Seconds(1))

ssc.checkpoint( "path")
val lines = ssc.socketTextStream("IP", PORT)
lines.countByValueAndWindow( Seconds(2), Seconds(8)).foreachRDD( s =>

{ println( "RDD ID IS : " + s.id) s.foreach( e => println("data is " + e._1 + " :" + e._2)) println() }

)

The result is wrong.
I checked the log, it showed:
17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Time 1498383086000 ms is valid
17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Window time = 2000 ms
17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Slide time = 8000 ms
17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Zero time = 1498383078000 ms
17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Current window = [1498383085000 ms, 1498383086000 ms]
17/06/25 17:31:26 DEBUG ReducedWindowedDStream: Previous window = [1498383077000 ms, 1498383078000 ms]
17/06/25 17:31:26 INFO ShuffledDStream: Slicing from 1498383077000 ms to 1498383084000 ms (aligned to 1498383077000 ms and 1498383084000 ms)
17/06/25 17:31:26 INFO ShuffledDStream: Time 1498383078000 ms is invalid as zeroTime is 1498383078000 ms , slideDuration is 1000 ms and difference is 0 ms
17/06/25 17:31:26 DEBUG ShuffledDStream: Time 1498383079000 ms is valid
17/06/25 17:31:26 DEBUG MappedDStream: Time 1498383079000 ms is valid

the slice time is wrong.

[BTW]: Team members,
If it was a bug, please don't fix it.I try to fix it myself.Thanks