Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-481

Support independent output committers for multiple outputs

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: Core
    • Labels:
      None

      Description

      I faced this issue while trying to write to Kite and HDFS in the same pipeline. A similar issue was logged for Kite[1][2].
      I was attempting to write a PCollection to Kite and a different PTable to HDFS as a text file. The write to Kite succeeded, however the write to HDFS only produced a _SUCCESS file with no text file.
      Commenting out the write to Kite seems to solve the issue and I can see the text file being written.

      [1] - https://issues.cloudera.org/browse/CDK-756
      [2] - http://mail-archives.apache.org/mod_mbox/crunch-dev/201401.mbox/%3CCAF-WD4QCUe0Toh3qewpDNnom3u786PVJLgH7T6Go_AbcTpLTaw@mail.gmail.com%3E

        Attachments

        1. CRUNCH-481.patch
          19 kB
          Tom White
        2. CRUNCH-481.patch
          17 kB
          Tom White
        3. CRUNCH-481.patch
          14 kB
          Tom White
        4. CRUNCH-481.patch
          18 kB
          Josh Wills
        5. CRUNCH-481c.patch
          17 kB
          Josh Wills
        6. CRUNCH-481-hadoop-2-compat.patch
          19 kB
          Ryan Brush

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              aniketk Aniket Kulkarni
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: