[SPARK-1681] Handle hive support correctly in ./make-distribution.sh

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Build, SQL
• Assignee: Patrick Wendell
• Reporter: Patrick Wendell
• Created: 30/Apr/14 06:37
• Updated: 02/Jul/14 09:45
• Resolved: 05/May/14 23:29

Description

When Hive support is enabled we should copy the datanucleus jars to the packaged distribution. The simplest way would be to create a lib_managed folder in the final distribution so that the compute-classpath script searches in exactly the same way whether or not it's a release.

A slightly nicer solution is to put the jars inside of `/lib` and have some fancier check for the jar location in the compute-classpath script.

We should also document how to run Spark SQL on YARN when hive support is enabled. In particular how to add the necessary jars to spark-submit.

Issue Links

• blocks: SPARK-1644 The org.datanucleus:* should not be packaged into spark-assembly-*.jar
• is related to: SPARK-1698 Improve spark integration