[OOZIE-1500] Fix many OS-specific issues on Windows

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.1.0
• Component/s: build, core, examples, scripts, tests

Environment

Windows

• Assignee: David Wannemacher
• Reporter: David Wannemacher
• Created: 18/Aug/13 16:04
• Updated: 09/Dec/14 00:45
• Resolved: 30/Sep/13 23:02

Description

This is a large patch, please let me know if I should split it up and/or use the code reviewer tool. I tested it on linux and verified that it did not cause any unit test regressions.

This patch fixes many unix-specific assumptions in code and tests:

1. Using "/" as a path separator. On windows, the right separator is "\". Fix: create a new Path object with the directory and subdirectory names, which handles using the right separator.
2. Determining whether a path is absolute by testing if it begins with "/". Fix: Create a new File object and use isAbsolute().
3. Generating a pathname by concatenating "file://", getTestCaseDir(), File.separator, and the file name. This is wrong because this is a URI, and File.separator is "/" on unix but "\" on windows. Fix: New helper method getTestCaseFileUri solves this.
4. Create directories by using "mkdir" shell command with hard-coded file separators. Fix: Create a new File object and use mkdirs().
5. Scripts that use unix-only commands (ls, echo, etc.). Fix: Implement windows-specific commands and use Shell.WINDOWS to pick the right one at runtime.
6. Assume that perl exists in TestShellActionExecutor.testPerlScript. It doesn't on Windows. Fix: skip the test on Windows.
7. Use the test case name as part of the temp directory name in order to ensure directories are unique. This can cause issues on Windows. Fix: Use a random UUID instead.
8. As a result of using getTestCaseUri, some test case verification that uses string equality fails. (TestLiteWorkflowAppService) Fix: use URI equivalence instead of string equality.
9. AdminServlet tests verify that system variables are working by checking for a USER variable. This is called USERNAME in windows. Fix: On Windows, check for USERNAME instead.
10. TestHostnameFilter checks that "localhost" is in the filter. On Windows, this is "127.0.0.1". Fix: On Windows, check for 127.0.0.1 instead.

On windows there are still several tests failing. I hope to spend some time fixing these later:

Failed tests:
testActionCheck(org.apache.oozie.command.wf.TestActionCheckXCommand)
testActionCheckTransientDuringLauncher(org.apache.oozie.command.wf.TestActionCheckXCommand)
testActionCheckTransientDuringMRAction(org.apache.oozie.command.wf.TestActionCheckXCommand)
testCoordinatorActionCommands(org.apache.oozie.sla.TestSLAEventGeneration): expected:<NOT_STARTED> but was:<IN_PROCESS>
testChmodWithGlob(org.apache.oozie.action.hadoop.TestFsActionExecutor): expected:<rw[-]------> but was:<rw[x]------>
Tests in error:
testWaitFor(org.apache.oozie.test.TestXTestCase): could not delete path [D:\apache\oozie\core\target\test-data\oozietests\org.apache.oozie.test.TestXTestCase$MyXTestCase\testWaitFor\conf\oozie-site.xml]
testBaseDir(org.apache.oozie.test.TestXTestCase): could not delete path [D:\apache\oozie\core\target\test-data\oozietests\org.apache.oozie.test.TestXTestCase$MyXTestCase\testBaseDir\conf\oozie-site.xml]

How to run unit tests in windows:

1. You will need to build hadoop-core from branch-1-win and install it into your local maven cache.
2. In the root pom.xml, replace the dummy value in HADOOP_HOME with the path to the hadoop directory. The minicluster tests require bin\winutils.exe to be in this directory.
3. In the root pom.xml (and hadooplibs version 1 pom.xmls, if OOZIE-1490 is not committed yet) change the hadoop version to the version built from branch-1-win.
4. May also need to use Hive 0.11 with the windows fixes.

Issue Links

• relates to: OOZIE-1640 Update documentation on delegation propagation