[SPARK-30576] Whether to block streaming batch commit, merge all blocking batchs as one batch commit - ASF JIRA

XML

Word

Printable

JSON

When the current job is not completed block streaming batch commit, until completed。The next job will merge all batch which during the blocking.

for example:

the input seq [1, 2, 3, 4, 5, 6]。

batch duration: 1s。

The 3th batch will take a long time。Normally the other batches will be completed quickly.

We expect:

1. the 4th batch will not be commited during the 3th batch computing, and 4th batch will be merge in the next batch. So that

the size of jobSets is always less than 1。

2. the num completedBatches less than the size of seq。

3. the data is not lost

links to

block streaming batch commit, until completed