Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-479

Writing to target with WriteMode.APPEND merges values into PCollection

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: Core
    • Labels:
      None

      Description

      This was mentioned as part of CDK-617[1]. A PCollection that contains a set of values, is written to a target with WriteMode.APPEND, and then that PCollection is materialized, when you iterate over that PCollection it contains not only the new values that were appended but also the existing values. This is surprising as most would expect that collection to only contain the original collection of values. A use case for this might be if the solution is looking to only process the new values instead of dealing with all of the existing data.

      [1] - https://issues.cloudera.org/browse/CDK-671

        Attachments

        1. CRUNCH-479b.patch
          14 kB
          Josh Wills
        2. CRUNCH-479.patch
          7 kB
          Micah Whitacre

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              mkwhitacre Micah Whitacre
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: