Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
There is an issue where the AvroPathPerKeyTarget won't properly copy the output of a Spark pipeline from the temp directory to the target directory because it assumes it will always get a valid Crunch output index (0, 1, 2, ...) and the SparkRuntime passes -1 to the Target's output handler method (to signal that it's the only output for the job.) I think the right move is to have AvroPathPerKeyTarget rewrite a -1 index to 0 so as not to break any other implementations that depend on the SparkRuntime's behavior.