[PIG-4854] Merge spark branch to trunk

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Unassigned
• Reporter: Pallavi Rao
• Created: 29/Mar/16 05:19
• Updated: 04/May/17 01:34
• Resolved: 04/May/17 01:34

Description

Believe the spark branch will be shortly ready to be merged with the main branch (couple of minor patches pending commit), given that we have addressed most functionality gaps and have ensured the UTs are clean. There are a few optimizations which we will take up once the branch is merged to trunk.

xuefuz, rohini, daijy,
Hopefully, you agree that the spark branch is ready for merge. If yes, how would like us to go about it? Do you want me to upload a huge patch that will be merged like any other patch or do you prefer a branch merge?

Issue Links

• links to: Review Board

Sub-Tasks

• Avoid add all spark dependency jars to SPARK_YARN_DIST_FILES and SPARK_DIST_CLASSPATH: Resolved
• Test spark branch with hadoop 1.x: Resolved
• Upgrade spark.version to 1.6.1: Resolved
• Task deserialization time is too long for spark on yarn mode: Closed
• Modify the content of start.xml for spark mode: Patch Available
• Support stop.on.failure in spark mode: Resolved
• TestRank3#testRankWithSplitInMap hangs after upgrade to spark 1.6.1: Closed
• Fix NPE exception in TestCustomPartitioner#testCustomPartitionerParseJoins: Resolved