Hadoop Common
  1. Hadoop Common
  2. HADOOP-10775

Shell operations to fail with meaningful errors on windows if winutils.exe not found

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: trunk-win
    • Fix Version/s: None
    • Component/s: util
    • Labels:
      None
    • Environment:

      Apache jenkins windows1 server

      Description

      If winutils.exe can't be found HADOOP_HOME wrong/unset or other causes, then an error is logged -but when any of the Shell operations are used, an NPE is raised rather than something meaningful.

      The error message at setup time should be preserved and then raised before any attempt to invoke a winutils-driven process made

        Issue Links

          Activity

          Hide
          Steve Loughran added a comment -

          Also, if winutils.exe is on the path, no check is actually made for this. the code in hadoop assumes

          1. that HADOOP_HOME is defined and exists
          2. that $HADOOP_HOME/bin/winutils.exe exists
          Show
          Steve Loughran added a comment - Also, if winutils.exe is on the path, no check is actually made for this. the code in hadoop assumes that HADOOP_HOME is defined and exists that $HADOOP_HOME/bin/winutils.exe exists
          Hide
          Steve Loughran added a comment -

          Example log with stack trace; the minicluster is trying to create a symlink by executing the command returned by Shell.getSymlinkCommand(targetPath, linkPath))

          java.lang.NullPointerException: null
          	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
          	at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
          	at org.apache.hadoop.util.Shell.run(Shell.java:418)
          	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
          	at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:171)
          	at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:197)
          	at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:209)
          	at org.apache.slider.test.YarnMiniClusterTestBase.createMiniCluster(YarnMiniClusterTestBase.groovy:241)
          	at org.apache.slider.test.YarnZKMiniClusterTestBase.createMiniCluster(YarnZKMiniClusterTestBase.groovy:105)
          	at org.apache.slider.test.YarnZKMiniClusterTestBase.createMiniCluster(YarnZKMiniClusterTestBase.groovy:124)
          	at org.apache.slider.agent.standalone.TestBadAMHeap.testBadAMHeap(TestBadAMHeap.groovy:43)Standard Output2014-07-02 13:44:56,336 [main] ERROR util.Shell (Shell.java:getWinUtilsPath(336)) - Failed to locate the winutils binary in the hadoop binary pathjava.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
          	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
          	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
          	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
          	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
          	at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:517)
          	at org.apache.slider.common.tools.SliderUtils.createConfiguration(SliderUtils.java:413)
          	at org.apache.slider.test.YarnMiniClusterTestBase.<clinit>(YarnMiniClusterTestBase.groovy:87)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:601)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          
          Show
          Steve Loughran added a comment - Example log with stack trace; the minicluster is trying to create a symlink by executing the command returned by Shell.getSymlinkCommand(targetPath, linkPath)) java.lang.NullPointerException: null at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010) at org.apache.hadoop.util.Shell.runCommand(Shell.java:445) at org.apache.hadoop.util.Shell.run(Shell.java:418) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650) at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:171) at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:197) at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:209) at org.apache.slider.test.YarnMiniClusterTestBase.createMiniCluster(YarnMiniClusterTestBase.groovy:241) at org.apache.slider.test.YarnZKMiniClusterTestBase.createMiniCluster(YarnZKMiniClusterTestBase.groovy:105) at org.apache.slider.test.YarnZKMiniClusterTestBase.createMiniCluster(YarnZKMiniClusterTestBase.groovy:124) at org.apache.slider.agent.standalone.TestBadAMHeap.testBadAMHeap(TestBadAMHeap.groovy:43)Standard Output2014-07-02 13:44:56,336 [main] ERROR util.Shell (Shell.java:getWinUtilsPath(336)) - Failed to locate the winutils binary in the hadoop binary pathjava.io.IOException: Could not locate executable null \bin\winutils.exe in the Hadoop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76) at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:517) at org.apache.slider.common.tools.SliderUtils.createConfiguration(SliderUtils.java:413) at org.apache.slider.test.YarnMiniClusterTestBase.<clinit>(YarnMiniClusterTestBase.groovy:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)

            People

            • Assignee:
              Unassigned
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development