[SPARK-28026] How to get the second row from 1 minute window

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.4.0
• Component/s: Examples, Structured Streaming
• Assignee: Unassigned
• Reporter: Devendra Vishwakarma
• Created: 12/Jun/19 12:12
• Updated: 12/Dec/22 18:10
• Resolved: 13/Jun/19 03:06

Description

I am almost blocked for a month I am still figuring out the API to achieve one of the functionalities related to spark structured streaming with window grouping.  So I thought to reach you guys here.

What I have is stock related time series data and I have grouped them in a 1-minute window along with the stock name. I am able to get first, last row in that 1-minute group, but I want some values from the second row of that 1 minute window, which I am not able to do at all. I looked at each function related to aggregation but I could not find any.Please help me.

This is what I have done so far -

val aggregates = stockEvents
.withWatermark("timestamp", "5 seconds")
.groupBy(window($"timestamp", "1 minute", "1 minute", "0 seconds"), $"stockName")
.agg(
first("tradingprice").alias("open"), //I have to make this value coming from second row
last("tradingprice").alias("close"),
max("tradingprice").alias("high"),
min("tradingprice").alias("low"))