[SPARK-20714] Fix match error when watermark is set with timeout = no timeout / processing timeout

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: Structured Streaming
• Target Version/s: 2.2.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 11/May/17 22:12
• Updated: 12/May/17 17:50
• Resolved: 12/May/17 17:50

Description

When watermark is set, and timeout conf is NoTimeout or ProcessingTimeTimeout (both do not need the watermark), the query fails at runtime with the following exception.

MatchException: Some(org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificPredicate@1a9b798e) (of class scala.Some)
    org.apache.spark.sql.execution.streaming.FlatMapGroupsWithStateExec$$anonfun$doExecute$1.apply(FlatMapGroupsWithStateExec.scala:120)
    org.apache.spark.sql.execution.streaming.FlatMapGroupsWithStateExec$$anonfun$doExecute$1.apply(FlatMapGroupsWithStateExec.scala:116)
    org.apache.spark.sql.execution.streaming.state.package$StateStoreOps$$anonfun$1.apply(package.scala:70)
    org.apache.spark.sql.execution.streaming.state.package$StateStoreOps$$anonfun$1.apply(package.scala:65)
    org.apache.spark.sql.execution.streaming.state.StateStoreRDD.compute(StateStoreRDD.scala:64)

Issue Links

• links to: [Github] Pull Request #17954 (tdas)