Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-647

Fault-tolerant sideInputs via Broadcast variables

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 0.5.0
    • runner-spark
    • None

    Description

      Following https://github.com/apache/incubator-beam/pull/909 which enables checkpointing to recover from failures, sideInputs (being implemented by broadcast variables) should be handled in a specific manner as described here: http://spark.apache.org/docs/1.6.2/streaming-programming-guide.html#accumulators-and-broadcast-variables.

      This is a bit more complicated than Aggregators (via Accumulators) as they are implemented using a single "aggregating" Accumulator, while a pipeline may contain multiple sideInputs.

      Attachments

        Activity

          People

            ksalant Kobi Salant
            amitsela Amit Sela
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: