Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28026

How to get the second row from 1 minute window

    XMLWordPrintableJSON

    Details

    • Type: Question
    • Status: Resolved
    • Priority: Major
    • Resolution: Invalid
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Labels:
      None

      Description

      I am almost blocked for a month I am still figuring out the API to achieve one of the functionalities related to spark structured streaming with window grouping.  So I thought to reach you guys here.

      What I have is stock related time series data and I have grouped them in a 1-minute window along with the stock name. I am able to get first, last row in that 1-minute group, but I want some values from the second row of that 1 minute window, which I am not able to do at all. I looked at each function related to aggregation but I could not find any.Please help me.

      This is what I have done so far -

      val aggregates = stockEvents
      .withWatermark("timestamp", "5 seconds")
      .groupBy(window($"timestamp", "1 minute", "1 minute", "0 seconds"), $"stockName")
      .agg(
      first("tradingprice").alias("open"), //I have to make this value coming from second row
      last("tradingprice").alias("close"),
      max("tradingprice").alias("high"),
      min("tradingprice").alias("low"))

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              vvvvvdev Devendra Vishwakarma
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: