[MAPREDUCE-4401] Enhancements to MapReduce for Windows Server and Windows Azure development and runtime environments

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Component/s: build, mrv2
• Tags: Windows
• Assignee: Bikas Saha
• Reporter: Bikas Saha
• Created: 05/Jul/12 17:59
• Updated: 12/May/16 18:24
• Resolved: 11/Feb/14 17:54

Description

This JIRA tracks the work that needs to be done on trunk to enable Hadoop to run on Windows Server and Azure environments. This incorporates porting relevant work from the similar effort on branch 1 tracked via HADOOP-8079.

Issue Links

• incorporates: MAPREDUCE-4959 bundling classpath into jar manifest on Windows does not expand environment variables or wildcards
• incorporates: MAPREDUCE-5331 hadoop pipes not working on Windows
• incorporates: MAPREDUCE-4739 Some MapReduce tests fail to find winutils.
• incorporates: MAPREDUCE-4780 MapReduce distribution build fails on Windows
• incorporates: MAPREDUCE-4869 TestMapReduceChildJVM fails in branch-trunk-win
• incorporates: MAPREDUCE-4870 TestMRJobsWithHistoryService causes infinite loop if it fails
• incorporates: MAPREDUCE-4872 MapReduce JVM launch does not use correct working directory on Windows
• incorporates: MAPREDUCE-4983 multiple MapReduce tests fail on Windows due to platform-specific assumptions in test code
• incorporates: MAPREDUCE-5080 TestContainerLauncher fails on Windows due to assertion failures on event/thread counts
• incorporates: MAPREDUCE-4885 Streaming tests have multiple failures on Windows
• incorporates: MAPREDUCE-4987 TestMRJobs#testDistributedCache fails on Windows due to classpath problems and unexpected behavior of symlinks
• incorporates: MAPREDUCE-5078 TestMRAppMaster fails on Windows due to mismatched path separators
• incorporates: MAPREDUCE-5187 Create mapreduce command scripts on Windows
• incorporates: MAPREDUCE-5259 TestTaskLog fails on Windows because of path separators missmatch
• incorporates: MAPREDUCE-5442 $HADOOP_MAPRED_HOME/$HADOOP_CONF_DIR setting not working on Windows
• incorporates: MAPREDUCE-5451 MR uses LD_LIBRARY_PATH which doesn't mean anything in Windows
• incorporates: MAPREDUCE-5546 mapred.cmd on Windows set HADOOP_OPTS incorrectly
• incorporates: MAPREDUCE-4374 Fix child task environment variable config and add support for Windows
• incorporates: MAPREDUCE-5349 TestClusterMapReduceTestCase and TestJobName fail on Windows in branch-2
• incorporates: MAPREDUCE-5355 MiniMRYarnCluster with localFs does not work on Windows
• incorporates: MAPREDUCE-5357 Job staging directory owner checking could fail on Windows
• incorporates: MAPREDUCE-5359 JobHistory should not use File.separator to match timestamp in path
• incorporates: MAPREDUCE-5360 TestMRJobClient fails on Windows due to path format
• incorporates: MAPREDUCE-5366 TestMRAsyncDiskService fails on Windows
• incorporates: MAPREDUCE-5161 Merge MAPREDUCE-1806 from branch-1 to branch-1-win. CombineFileInputFormat fix for paths not on default FS
• incorporates: MAPREDUCE-5045 UtilTest#isCygwin method appears to be unused
• is part of: HADOOP-8562 Enhancements to support Hadoop on Windows Server and Windows Azure environments
• is related to: MAPREDUCE-5391 TestNonLocalJobJarSubmission fails on Windows due to missing classpath entries
• is related to: MAPREDUCE-5075 DistCp leaks input file handles
• is related to: MAPREDUCE-5177 Move to common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
• relates to: MAPREDUCE-5056 TestProcfsBasedProcessTree fails on Windows with Process-tree dump doesn't start with a proper header
• relates to: MAPREDUCE-5191 TestQueue#testQueue fails with timeout on Windows