Hive
  1. Hive
  2. HIVE-5518

ADD JAR should add entries to local classpath

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.12.0
    • Fix Version/s: None
    • Component/s: CLI
    • Labels:
      None

      Description

      Jars referenced in ADD JAR statements are not made available on the immediate classpath. That means they're useless for scripts which need to initialize external output formats for job submission (ie, hbase storage handler). Is this expected behavior?

      For example, the table 'pagecounts_hbase' is an hbase table defined using the HBaseStorageHandler

      $ cat foo.hql
      ADD FILE /etc/hbase/conf/hbase-site.xml;
      ADD JAR /usr/lib/hbase/lib/hbase-common-0.96.0.2.0.6.0-68-hadoop2.jar;
      ADD JAR /usr/lib/hbase/lib/hbase-server-0.96.0.2.0.6.0-68-hadoop2.jar;
      ADD JAR /usr/lib/hbase/lib/hbase-client-0.96.0.2.0.6.0-68-hadoop2.jar;
      ADD JAR /usr/lib/hbase/lib/hbase-protocol-0.96.0.2.0.6.0-68-hadoop2.jar;
      
      FROM pgc INSERT INTO TABLE pagecounts_hbase SELECT pgc.* WHERE rowkey LIKE 'en/q%' LIMIT 10;
      $ hive -f foo.hql
      ...
      Added resource: /etc/hbase/conf/hbase-site.xml
      Added /usr/lib/hbase/lib/hbase-common-0.96.0.2.0.6.0-68-hadoop2.jar to class path
      Added resource: /usr/lib/hbase/lib/hbase-common-0.96.0.2.0.6.0-68-hadoop2.jar
      ...
      Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/mapreduce/TableInputFormatBase                                                                                [29/1858]
              at java.lang.ClassLoader.defineClass1(Native Method)
              at java.lang.ClassLoader.defineClass(ClassLoader.java:791)
              at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
              at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
              at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
              at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
              at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:410)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
              at java.lang.Class.forName0(Native Method)
              at java.lang.Class.forName(Class.java:266)
              at org.apache.hadoop.hive.ql.metadata.Table.getInputFormatClass(Table.java:305)
              at org.apache.hadoop.hive.ql.metadata.Table.<init>(Table.java:98)
              at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:989)
              at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:892)
              at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$tableSpec.<init>(BaseSemanticAnalyzer.java:730)
              at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer$tableSpec.<init>(BaseSemanticAnalyzer.java:707)
              at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1196)
              at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1053)
              at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8342)
              at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:284)
              at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:441)
              at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:342)
              at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:977)
              at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
              at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
              at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
              at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
              at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
              at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:446)
              at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:456)
              at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:737)
              at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
              at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:601)
              at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.mapreduce.TableInputFormatBase
              at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
              at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:423)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:356)
              ... 48 more
      

      The ADDed jar hbase-server.jar contains the missing class:

      $ unzip -l /usr/lib/hbase/lib/hbase-server-0.96.0.2.0.6.0-68-hadoop2.jar | grep TableInputFormatBase
           5363  10-09-2013 19:45   org/apache/hadoop/hbase/mapred/TableInputFormatBase.class
           7460  10-09-2013 19:45   org/apache/hadoop/hbase/mapreduce/MultiTableInputFormatBase.class
           8803  10-09-2013 19:45   org/apache/hadoop/hbase/mapreduce/TableInputFormatBase.class
      

        Activity

          People

          • Assignee:
            Unassigned
            Reporter:
            Nick Dimiduk
          • Votes:
            0 Vote for this issue
            Watchers:
            9 Start watching this issue

            Dates

            • Created:
              Updated:

              Development