[HIVE-12179] Add option to not add spark-assembly.jar to Hive classpath

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.3.0, 2.0.0
• Component/s: Spark
• Assignee: Jason Dere
• Reporter: Jason Dere
• Created: 14/Oct/15 20:30
• Updated: 16/Feb/16 23:52
• Resolved: 21/Oct/15 18:00

Description

After running the following Hive script:

add jar hdfs:///tmp/junit-4.11.jar;
show tables;

I can see the following lines getting printed to stdout when Hive exits:

WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked.
WARN: Please see http://www.slf4j.org/codes.html#release for an explanation.

Also seeing the following warnings in stderr:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.3.0-2981/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.3.0-2981/spark/lib/spark-assembly-1.4.1.2.3.3.0-2981-hadoop2.7.1.2.3.3.0-2981.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.3.0-2981/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.3.3.0-2981/spark/lib/spark-assembly-1.4.1.2.3.3.0-2981-hadoop2.7.1.2.3.3.0-2981.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

It looks like this is due to the addition of the shaded spark-assembly.jar to the classpath, which contains classes from icl-over-slf4j.jar (which is causing the stdout messages) and slf4j-log4j12.jar.

Removing spark-assembly.jar from being added to the classpath causes these messages to go away. It would be good to have a way to specify that Hive not add spark-assembly.jar to the class path.

Issue Links

• relates to: HIVE-12880 spark-assembly causes Hive class version problems