Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1177

Input DStream "bundles" should be in serialized form and include relevant metadata.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 0.5.0
    • runner-spark
    • None

    Description

      Currently, the input partitions hold "bundles" of read elements within the mapWithStateDStream used for the read.
      Since this is automatically shuffled, user-data (the read elements) should be serialized using coders to avoid breaking (if user-data is not Kryo serializable).
      Even after BEAM-848 would complete, the resulting MapWithStateDStream would be checkpointed periodically and so it would still have to remain in serialized form.

      Attachments

        Issue Links

          Activity

            People

              amitsela Amit Sela
              amitsela Amit Sela
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: