[PIG-4746] Ensure spark can be run as PIG action in Oozie

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Prateek Vaishnav
• Reporter: Pallavi Rao
• Created: 27/Nov/15 07:24
• Updated: 04/Nov/16 03:14

Description

I was able get PIG on SPARK going with Oozie. But, only in "local" mode. Here is what I did:
1. Used workflow schema version uri:oozie:workflow:0.2 and passed exectype as an argument.
2. Copied Spark jars + kyro jar into workflow app lib.

To get spork going in yarn-client mode, couple of enhancements will need to be made:
1. Right now, spark launcher reads SPARK_MASTER as env. variable. Need to make this a PIG property.
2. The spark libraries need to be in classpath of the driver in case of yarn-clientmode. This will need a fix similar to PIG-4667