[SPARK-6847] Stack overflow on updateStateByKey which followed by a dstream with checkpoint set

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.0, 1.4.1, 1.5.2, 1.6.0
• Fix Version/s: 2.0.0
• Component/s: DStreams
• Labels: StackOverflowError Streaming
• Target Version/s: 2.0.0
• Assignee: Shixiong Zhu
• Reporter: Jack Hu
• Created: 10/Apr/15 09:50
• Updated: 01/Feb/16 20:05
• Resolved: 01/Feb/16 19:02

Description

The issue happens with the following sample code: uses updateStateByKey followed by a map with checkpoint interval 10 seconds

    val sparkConf = new SparkConf().setAppName("test")
    val streamingContext = new StreamingContext(sparkConf, Seconds(10))
    streamingContext.checkpoint("""checkpoint""")
    val source = streamingContext.socketTextStream("localhost", 9999)
    val updatedResult = source.map(
        (1,_)).updateStateByKey(
            (newlist : Seq[String], oldstate : Option[String]) =>     newlist.headOption.orElse(oldstate))
    updatedResult.map(_._2)
    .checkpoint(Seconds(10))
    .foreachRDD((rdd, t) => {
      println("Deep: " + rdd.toDebugString.split("\n").length)
      println(t.toString() + ": " + rdd.collect.length)
    })
    streamingContext.start()
    streamingContext.awaitTermination()

From the output, we can see that the dependency will be increasing time over time, the updateStateByKey never get check-pointed, and finally, the stack overflow will happen.

Note:

• The rdd in updatedResult.map(_._2) get check-pointed in this case, but not the updateStateByKey
• If remove the checkpoint(Seconds(10)) from the map result ( updatedResult.map(_._2) ), the stack overflow will not happen

Issue Links

• links to: [Github] Pull Request #10934 (zsxwing)