[BEAM-648] Persist and restore Aggergator values in case of recovery from failure

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 0.6.0
• Component/s: runner-spark
• Assignee: Aviem Zur
• Reporter: Amit Sela
• Created: 20/Sep/16 18:58
• Updated: 16/May/20 13:38
• Resolved: 30/Jan/17 21:00

Description

While Aggregators are fault-tolerant (pending https://github.com/apache/incubator-beam/pull/909), they will recover into their initial value.

The SparkRunner should persist the values to durable storage and recover from the persisted state.

One way to go would be to persist via Spark's listeners, for example "onBatchCompleted".

For more details see SPARK-5206.

Issue Links

• links to: GitHub Pull Request #1815