[SPARK-22230] agg(last('attr)) gives weird results for streaming

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Structured Streaming
• Assignee: Jose Torres
• Reporter: Jose Torres
• Created: 09/Oct/17 20:54
• Updated: 10/Oct/17 00:05
• Resolved: 10/Oct/17 00:04

Description

In stream aggregation, last('attr) yields the last value from the first microbatch forever. I'm not sure if it's fair to call this a correctness bug, since last doesn't have strong correctness semantics, but ignoring all rows past the first microbatch is at least weird.

Simple repro in StreamingAggregationSuite:

val input = MemoryStream[Int]

val aggregated = input.toDF().agg(last('value))
testStream(aggregated, OutputMode.Complete())(
AddData(input, 1, 2, 3),
CheckLastBatch(3),
AddData(input, 4, 5, 6),
CheckLastBatch(6) // actually yields 3 again

Issue Links

• links to: [Github] Pull Request #19461 (joseph-torres)