Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-14942

Reduce delay between batch construction and execution

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.0.0
    • SQL
    • None

    Description

      Currently in StreamExecution, we first run the batch, then construct the next:

      if (dataAvailable) runBatch()
      constructNextBatch()
      

      This is good if we run batches ASAP, where data would get processed in the very next batch:

      However, if we run batches at trigger like ProcessTime("1 minute"), data - such as y below - may not get processed in the very next batch i.e. batch 1, but in batch 2:

      Attachments

        Activity

          People

            proflin Liwei Lin(Inactive)
            proflin Liwei Lin(Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: