Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12179

Add option to not add spark-assembly.jar to Hive classpath

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0, 2.0.0
    • Component/s: Spark
    • Labels:
      None

      Description

      After running the following Hive script:

      add jar hdfs:///tmp/junit-4.11.jar;
      show tables;
      

      I can see the following lines getting printed to stdout when Hive exits:

      WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked.
      WARN: Please see http://www.slf4j.org/codes.html#release for an explanation.
      

      Also seeing the following warnings in stderr:

      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/usr/hdp/2.3.3.0-2981/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/hdp/2.3.3.0-2981/spark/lib/spark-assembly-1.4.1.2.3.3.0-2981-hadoop2.7.1.2.3.3.0-2981.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      SLF4J: Class path contains multiple SLF4J bindings.
      SLF4J: Found binding in [jar:file:/usr/hdp/2.3.3.0-2981/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: Found binding in [jar:file:/usr/hdp/2.3.3.0-2981/spark/lib/spark-assembly-1.4.1.2.3.3.0-2981-hadoop2.7.1.2.3.3.0-2981.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      

      It looks like this is due to the addition of the shaded spark-assembly.jar to the classpath, which contains classes from icl-over-slf4j.jar (which is causing the stdout messages) and slf4j-log4j12.jar.

      Removing spark-assembly.jar from being added to the classpath causes these messages to go away. It would be good to have a way to specify that Hive not add spark-assembly.jar to the class path.

        Attachments

        1. HIVE-12179.1.patch
          0.4 kB
          Jason Dere
        2. HIVE-12179.2.patch
          0.4 kB
          Jason Dere

          Issue Links

            Activity

              People

              • Assignee:
                jdere Jason Dere
                Reporter:
                jdere Jason Dere
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: