Uploaded image for project: 'Crunch'
  1. Crunch
  2. CRUNCH-670

Make the AvroPathPerKeyTarget work with the SparkRuntime

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      There is an issue where the AvroPathPerKeyTarget won't properly copy the output of a Spark pipeline from the temp directory to the target directory because it assumes it will always get a valid Crunch output index (0, 1, 2, ...) and the SparkRuntime passes -1 to the Target's output handler method (to signal that it's the only output for the job.) I think the right move is to have AvroPathPerKeyTarget rewrite a -1 index to 0 so as not to break any other implementations that depend on the SparkRuntime's behavior.

        Attachments

          Activity

            People

            • Assignee:
              jwills Josh Wills
              Reporter:
              jwills Josh Wills
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: