[HADOOP-10775] Shell operations to fail with meaningful errors on windows if winutils.exe not found

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: trunk-win, 2.7.1
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: util

Environment

windows

• Target Version/s: 2.8.0
• Hadoop Flags: Reviewed
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 02/Jul/14 13:17
• Updated: 29/Aug/19 17:38
• Resolved: 13/Oct/15 20:30

Description

If winutils.exe can't be found HADOOP_HOME wrong/unset or other causes, then an error is logged -but when any of the Shell operations are used, an NPE is raised rather than something meaningful.

The error message at setup time should be preserved and then raised before any attempt to invoke a winutils-driven process made

Issue Links

• breaks: HADOOP-12478 Shell.getWinUtilsPath() has been renamed Shell.getWinutilsPath()
• is duplicated by: SPARK-2356 Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
• is duplicated by: HADOOP-11003 org.apache.hadoop.util.Shell should not take a dependency on binaries being deployed when used as a library
• is duplicated by: HADOOP-14586 StringIndexOutOfBoundsException breaks org.apache.hadoop.util.Shell on 2.7.x with Java 9
• is related to: SPARK-6961 Cannot save data to parquet files when executing from Windows from a Maven Project
• relates to: SLIDER-201 jenkins to add windows builds
• relates to: SPARK-2356 Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
• relates to: HADOOP-11003 org.apache.hadoop.util.Shell should not take a dependency on binaries being deployed when used as a library
• relates to: CALCITE-3307 PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows