Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30576

Whether to block streaming batch commit, merge all blocking batchs as one batch commit

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 2.4.4
    • Fix Version/s: None
    • Component/s: DStreams
    • Labels:

      Description

      When the current job is not completed block streaming batch commit, until completed。The next job will merge all batch which  during the blocking.

      for example:

       the input seq [1, 2, 3, 4, 5, 6]。 

      batch duration: 1s。

      The 3th batch will take a long time。Normally the other batches will be completed quickly.

      We expect:

            1. the 4th batch will not be commited during the 3th batch computing, and 4th batch will be merge in the next batch. So that

      the size of jobSets  is always less than 1。

            2. the num completedBatches less than the size of seq。

            3. the data is not lost

       

       

        Attachments

          Activity

            People

            • Assignee:
              apachespark Apache Spark
              Reporter:
              yingshikong186 liushuo
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: