[PIG-4549] Set CROSS operation parallelism for Spark engine

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: spark-branch
• Fix Version/s: spark-branch
• Component/s: spark
• Assignee: Mohit Sabharwal
• Reporter: Mohit Sabharwal
• Created: 13/May/15 01:48
• Updated: 21/Jun/17 09:18
• Resolved: 22/May/15 18:05

Description

Spark engine should set parallelism to be used for CROSS operation by GFCross UDF.

If not set, GFCross throws an exception:

                String s = cfg.get(PigImplConstants.PIG_CROSS_PARALLELISM + "." + crossKey);
                if (s == null) {
                    throw new IOException("Unable to get parallelism hint from job conf");
                }

Estimating parallelism for Spark engine is a TBD item. Until that is done, for CROSS to work, we should use the default parallelism value in GFCross.

Issue Links

• blocks: PIG-4189 Make cross join work with Spark
• links to: review board