[CRUNCH-509] Crunch with Spark doesn't name all outputs

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.11.0
• Fix Version/s: 0.12.0
• Component/s: Core
• Assignee: Josh Wills
• Reporter: Micah Whitacre
• Created: 07/Apr/15 19:51
• Updated: 18/May/15 19:04
• Resolved: 08/May/15 21:53

Description

Crunch currently does not "name" all outputs when running with a SparkPipeline. This becomes a problem as some Targets (based on CRUNCH-82) have coded in checked to ensure that the name must be populated. Specifically the implementation I'm running into issues with is the Kite DatasetTarget[2].

Need to read up a bit on context to see if it is a Crunch/Kite issue or where it is easiest/correct to fix. jwills or tomwhite feedback would be welcome.

[1] - https://github.com/apache/crunch/blob/3ab0b078c47f23b3ba893fdfb05fd723f663d02b/crunch-spark/src/main/java/org/apache/crunch/impl/spark/SparkRuntime.java#L337
[2] - https://github.com/kite-sdk/kite/blob/e080f0237e7383a16fff8547ad43387ccf55c473/kite-data/kite-data-crunch/src/main/java/org/kitesdk/data/crunch/DatasetTarget.java#L178