Hadoop Common
  1. Hadoop Common
  2. HADOOP-10775

Shell operations to fail with meaningful errors on windows if winutils.exe not found

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: trunk-win
    • Fix Version/s: None
    • Component/s: util
    • Labels:
      None
    • Environment:

      Apache jenkins windows1 server

      Description

      If winutils.exe can't be found HADOOP_HOME wrong/unset or other causes, then an error is logged -but when any of the Shell operations are used, an NPE is raised rather than something meaningful.

      The error message at setup time should be preserved and then raised before any attempt to invoke a winutils-driven process made

        Issue Links

          Activity

          Steve Loughran created issue -
          Hide
          Steve Loughran added a comment -

          Example log with stack trace; the minicluster is trying to create a symlink by executing the command returned by Shell.getSymlinkCommand(targetPath, linkPath))

          java.lang.NullPointerException: null
          	at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
          	at org.apache.hadoop.util.Shell.runCommand(Shell.java:445)
          	at org.apache.hadoop.util.Shell.run(Shell.java:418)
          	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
          	at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:171)
          	at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:197)
          	at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:209)
          	at org.apache.slider.test.YarnMiniClusterTestBase.createMiniCluster(YarnMiniClusterTestBase.groovy:241)
          	at org.apache.slider.test.YarnZKMiniClusterTestBase.createMiniCluster(YarnZKMiniClusterTestBase.groovy:105)
          	at org.apache.slider.test.YarnZKMiniClusterTestBase.createMiniCluster(YarnZKMiniClusterTestBase.groovy:124)
          	at org.apache.slider.agent.standalone.TestBadAMHeap.testBadAMHeap(TestBadAMHeap.groovy:43)Standard Output2014-07-02 13:44:56,336 [main] ERROR util.Shell (Shell.java:getWinUtilsPath(336)) - Failed to locate the winutils binary in the hadoop binary pathjava.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
          	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
          	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
          	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
          	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
          	at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:517)
          	at org.apache.slider.common.tools.SliderUtils.createConfiguration(SliderUtils.java:413)
          	at org.apache.slider.test.YarnMiniClusterTestBase.<clinit>(YarnMiniClusterTestBase.groovy:87)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:601)
          	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
          	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
          	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
          	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
          	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
          	at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
          	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          
          Show
          Steve Loughran added a comment - Example log with stack trace; the minicluster is trying to create a symlink by executing the command returned by Shell.getSymlinkCommand(targetPath, linkPath)) java.lang.NullPointerException: null at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010) at org.apache.hadoop.util.Shell.runCommand(Shell.java:445) at org.apache.hadoop.util.Shell.run(Shell.java:418) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650) at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:171) at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:197) at org.apache.hadoop.yarn.server.MiniYARNCluster.<init>(MiniYARNCluster.java:209) at org.apache.slider.test.YarnMiniClusterTestBase.createMiniCluster(YarnMiniClusterTestBase.groovy:241) at org.apache.slider.test.YarnZKMiniClusterTestBase.createMiniCluster(YarnZKMiniClusterTestBase.groovy:105) at org.apache.slider.test.YarnZKMiniClusterTestBase.createMiniCluster(YarnZKMiniClusterTestBase.groovy:124) at org.apache.slider.agent.standalone.TestBadAMHeap.testBadAMHeap(TestBadAMHeap.groovy:43)Standard Output2014-07-02 13:44:56,336 [main] ERROR util.Shell (Shell.java:getWinUtilsPath(336)) - Failed to locate the winutils binary in the hadoop binary pathjava.io.IOException: Could not locate executable null \bin\winutils.exe in the Hadoop binaries. at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318) at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326) at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76) at org.apache.hadoop.yarn.conf.YarnConfiguration.<clinit>(YarnConfiguration.java:517) at org.apache.slider.common.tools.SliderUtils.createConfiguration(SliderUtils.java:413) at org.apache.slider.test.YarnMiniClusterTestBase.<clinit>(YarnMiniClusterTestBase.groovy:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
          Steve Loughran made changes -
          Field Original Value New Value
          Summary Shell operations could fail with meaningful errors on windows if winutils.exe not found Shell operations to fail with meaningful errors on windows if winutils.exe not found
          Steve Loughran made changes -
          Link This issue relates to SLIDER-201 [ SLIDER-201 ]
          Hide
          Steve Loughran added a comment -

          Also, if winutils.exe is on the path, no check is actually made for this. the code in hadoop assumes

          1. that HADOOP_HOME is defined and exists
          2. that $HADOOP_HOME/bin/winutils.exe exists
          Show
          Steve Loughran added a comment - Also, if winutils.exe is on the path, no check is actually made for this. the code in hadoop assumes that HADOOP_HOME is defined and exists that $HADOOP_HOME/bin/winutils.exe exists
          Steve Loughran made changes -
          Link This issue relates to SPARK-2356 [ SPARK-2356 ]
          Steve Loughran made changes -
          Link This issue relates to HADOOP-11003 [ HADOOP-11003 ]
          Steve Loughran made changes -
          Link This issue is related to SPARK-6961 [ SPARK-6961 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development