[HADOOP-11003] org.apache.hadoop.util.Shell should not take a dependency on binaries being deployed when used as a library

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: util

Environment

Windows

• Tags: windows, winutils
• Assignee: Steve Loughran
• Reporter: Remus Rusanu
• Created: 26/Aug/14 11:27
• Updated: 29/Aug/16 22:16
• Resolved: 13/Oct/15 20:32

Description

HIVE-7845 shows how an exception is being thrown when org.apache.hadoop.util.Shell is being used as a library, not as part of a deployed Hadoop environment.

13:20:00 [ERROR pool-2-thread-4 Shell.getWinUtilsPath] Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
               at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:324)
               at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:339)
               at org.apache.hadoop.util.Shell.<clinit>(Shell.java:332)
               at org.apache.hadoop.hive.conf.HiveConf$ConfVars.findHadoopBinary(HiveConf.java:918)
               at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:228)

There are similar native dependencies (eg. NativeIO and hadoop.dll) that handle lack of binaries with fallback to non-native code paths.

Issue Links

• duplicates: HADOOP-10775 Shell operations to fail with meaningful errors on windows if winutils.exe not found
• is related to: HADOOP-10051 winutil.exe is not included in hadoop bin tarball
• is related to: HADOOP-10775 Shell operations to fail with meaningful errors on windows if winutils.exe not found
• relates to: SPARK-2356 Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
• relates to: HIVE-7845 "Failed to locate the winutils binary" when loading JDBC driver on Windows