Uploaded image for project: 'Crunch (Retired)'
  1. Crunch (Retired)
  2. CRUNCH-479

Writing to target with WriteMode.APPEND merges values into PCollection

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.12.0
    • Core
    • None

    Description

      This was mentioned as part of CDK-617[1]. A PCollection that contains a set of values, is written to a target with WriteMode.APPEND, and then that PCollection is materialized, when you iterate over that PCollection it contains not only the new values that were appended but also the existing values. This is surprising as most would expect that collection to only contain the original collection of values. A use case for this might be if the solution is looking to only process the new values instead of dealing with all of the existing data.

      [1] - https://issues.cloudera.org/browse/CDK-671

      Attachments

        1. CRUNCH-479.patch
          7 kB
          Micah Whitacre
        2. CRUNCH-479b.patch
          14 kB
          Josh Wills

        Activity

          People

            jwills Josh Wills
            mkwhitacre Micah Whitacre
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment