[CRUNCH-670] Make the AvroPathPerKeyTarget work with the SparkRuntime

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Assignee: Josh Wills
• Reporter: Josh Wills
• Created: 24/May/18 05:45
• Updated: 23/Sep/19 18:32
• Resolved: 23/Sep/19 18:32

Description

There is an issue where the AvroPathPerKeyTarget won't properly copy the output of a Spark pipeline from the temp directory to the target directory because it assumes it will always get a valid Crunch output index (0, 1, 2, ...) and the SparkRuntime passes -1 to the Target's output handler method (to signal that it's the only output for the job.) I think the right move is to have AvroPathPerKeyTarget rewrite a -1 index to 0 so as not to break any other implementations that depend on the SparkRuntime's behavior.