[CALCITE-3307] PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.21.0
• Labels: pull-request-available
• Assignee: Ruben Q L
• Reporter: Julian Hyde
• Created: 29/Aug/19 17:37
• Updated: 10/Sep/19 21:36
• Resolved: 30/Aug/19 13:02

Description

Running PigRelExTest on Windows, after CALCITE-3122 was merged, gives the following error:

2019-08-29 15:33:23,229 [main] ERROR - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
	at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:382)
	at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:397)
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:390)
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80)
	at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
	at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
	at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
	at org.apache.hadoop.security.UserGroupInformation.isAuthenticationMethodEnabled(UserGroupInformation.java:338)
	at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:332)
	at org.apache.pig.backend.hadoop.HKerberos.tryKerberosKeytabLogin(HKerberos.java:70)
	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:220)
	at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.init(HExecutionEngine.java:112)
	at org.apache.pig.impl.PigContext.connect(PigContext.java:305)
	at org.apache.pig.PigServer.<init>(PigServer.java:231)
	at org.apache.pig.PigServer.<init>(PigServer.java:219)
	at org.apache.pig.PigServer.<init>(PigServer.java:211)
	at org.apache.pig.PigServer.<init>(PigServer.java:207)
	at org.apache.calcite.piglet.PigConverter.<init>(PigConverter.java:107)
	at org.apache.calcite.piglet.PigConverter.create(PigConverter.java:112)
	at org.apache.calcite.test.PigRelTestBase.testSetup(PigRelTestBase.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)

This issue has arisen many times; see HADOOP-10775 and SPARK-2356.

Issue Links

• is related to: SPARK-2356 Exception: Could not locate executable null\bin\winutils.exe in the Hadoop
• is related to: CALCITE-3122 Convert Pig Latin scripts into Calcite logical plans
• is related to: HADOOP-10775 Shell operations to fail with meaningful errors on windows if winutils.exe not found
• links to: GitHub Pull Request #1425