Hive
  1. Hive
  2. HIVE-6962

NoSuchMethodError exception when using HIVE 0.13 with Hadoop 1.0.4

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.13.0
    • Fix Version/s: None
    • Component/s: Query Processor
    • Labels:
      None
    • Environment:

      Hadoop 1.0.4

      Description

      Use HIVE 0.13 with Hadoop 1.0.4. When executing a query, get the following exception:

      Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.unset(Ljava/lang/String;)V
      	at org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3416)
      	at org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3401)
      	at org.apache.hadoop.hive.ql.Context.getScratchDir(Context.java:214)
      	at org.apache.hadoop.hive.ql.Context.getMRScratchDir(Context.java:265)
      	at org.apache.hadoop.hive.ql.Context.getMRTmpPath(Context.java:323)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1442)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:1192)
      	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9209)
      	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:327)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:391)
      	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:291)
      	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:944)
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1009)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:880)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:870)
      	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
      	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
      	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423)
      	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
      	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
      	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:597)
      	at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
      

      This is because there is no unset() method in some versions of Hadoop.

        Issue Links

          Activity

          Hide
          Thejas M Nair added a comment -

          Please not that with Hive 0.13.1 the workaround that I proposed above should work. (setting fs.permissions.umask-mode to some value).

          Yes, catching NoSuchMethodError as you suggest could be a fix. But it would be better to do such version specific logic in the shims. ie, add a HadoopShims functions for this, and do the catch only in Hadoop20SShims.

          Show
          Thejas M Nair added a comment - Please not that with Hive 0.13.1 the workaround that I proposed above should work. (setting fs.permissions.umask-mode to some value). Yes, catching NoSuchMethodError as you suggest could be a fix. But it would be better to do such version specific logic in the shims. ie, add a HadoopShims functions for this, and do the catch only in Hadoop20SShims.
          Hide
          Sun Rui added a comment -

          As mentioned above, JDBC through HS2 still has this issue. I workaround it by catching the exception:
          try

          { conf.unset("fs.permissions.umask-mode"); }

          catch (NoSuchMethodError e) {
          }
          Could this be a fix?

          Show
          Sun Rui added a comment - As mentioned above, JDBC through HS2 still has this issue. I workaround it by catching the exception: try { conf.unset("fs.permissions.umask-mode"); } catch (NoSuchMethodError e) { } Could this be a fix?
          Hide
          Thejas M Nair added a comment - - edited

          After changes in HIVE-7001, you would see this with pre hadoop 1.2 versions, only if hive.server2.enable.doAs=true (and ofcourse you are running the query through HS2). I hope to get HIVE-7001 into hive 0.13.1 .

          The workaround for those versions while using that combination would be to set fs.permissions.umask-mode as described above (Verified that this works).
          ie. add this in hive-site.xml -

            <property>
              <name>fs.permissions.umask-mode</name>
              <value>022</value>
              <description> Setting a value for fs.permissions.umask-mode to work around issue in HIVE-6962.
           It has no impact in hadoop 1.x line on hdfs operations.
              </description>
            </property>
          
          Show
          Thejas M Nair added a comment - - edited After changes in HIVE-7001 , you would see this with pre hadoop 1.2 versions, only if hive.server2.enable.doAs=true (and ofcourse you are running the query through HS2). I hope to get HIVE-7001 into hive 0.13.1 . The workaround for those versions while using that combination would be to set fs.permissions.umask-mode as described above (Verified that this works). ie. add this in hive-site.xml - <property> <name>fs.permissions.umask-mode</name> <value>022</value> <description> Setting a value for fs.permissions.umask-mode to work around issue in HIVE-6962. It has no impact in hadoop 1.x line on hdfs operations. </description> </property>
          Hide
          Sun Rui added a comment -

          Thejas M Nair
          Configuration.set(null) does not work:

          [main]: ERROR ql.Driver: FAILED: NullPointerException null
          java.lang.NullPointerException
          	at java.util.Hashtable.put(Hashtable.java:394)
          	at java.util.Properties.setProperty(Properties.java:143)
          	at org.apache.hadoop.conf.Configuration.set(Configuration.java:438)
          	at org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3417)
          

          I workarounded it simply by removing the line of code I haven't test your WA.

          Seems shims should be used to abstract such difference across Hadoop versions.

          Show
          Sun Rui added a comment - Thejas M Nair Configuration.set(null) does not work: [main]: ERROR ql.Driver: FAILED: NullPointerException null java.lang.NullPointerException at java.util.Hashtable.put(Hashtable.java:394) at java.util.Properties.setProperty(Properties.java:143) at org.apache.hadoop.conf.Configuration.set(Configuration.java:438) at org.apache.hadoop.hive.ql.exec.Utilities.createDirsWithPermission(Utilities.java:3417) I workarounded it simply by removing the line of code I haven't test your WA. Seems shims should be used to abstract such difference across Hadoop versions.
          Hide
          Thejas M Nair added a comment -

          I haven't tested this workaround, please update jira if it worked for you.

          Patches to hive are also welcome.

          Show
          Thejas M Nair added a comment - I haven't tested this workaround, please update jira if it worked for you. Patches to hive are also welcome.
          Hide
          Thejas M Nair added a comment - - edited

          Looks like this method Configuration.unset was added in hadoop 1.2 .
          Not sure if Configuration.set(null) can be used in place of unset.

          From the patch, it looks like a workaround would be to set "fs.permissions.umask-mode" to a value (say "022") in configuration (in hive-site.xml , hdfs-site or core-site). I just realized that this is a new parameter name in hadoop 2.x and does not exist in hadoop 1.x (at least not in hadoop 1.2). So setting this fs.permissions.umask-mode=022 will not have any side effects. AFAIK, that codepath is actually needed only if you have multiple subdirs in scratch dir that need to be created . So if you are using the default scratch dir in hive

          /tmp/hive-${username} 

          then this workaround would be safe.

          Show
          Thejas M Nair added a comment - - edited Looks like this method Configuration.unset was added in hadoop 1.2 . Not sure if Configuration.set(null) can be used in place of unset. From the patch, it looks like a workaround would be to set "fs.permissions.umask-mode" to a value (say "022") in configuration (in hive-site.xml , hdfs-site or core-site). I just realized that this is a new parameter name in hadoop 2.x and does not exist in hadoop 1.x (at least not in hadoop 1.2). So setting this fs.permissions.umask-mode=022 will not have any side effects. AFAIK, that codepath is actually needed only if you have multiple subdirs in scratch dir that need to be created . So if you are using the default scratch dir in hive /tmp/hive-${username} then this workaround would be safe.

            People

            • Assignee:
              Unassigned
              Reporter:
              Sun Rui
            • Votes:
              2 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Development