[YARN-191] Enhancements to YARN for Windows Server and Windows Azure development and runtime environments

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Assignee: Bikas Saha
• Reporter: Bikas Saha
• Created: 31/Oct/12 14:06
• Updated: 11/Feb/14 17:54
• Resolved: 11/Feb/14 17:54

Description

This JIRA tracks the work that needs to be done on trunk to enable Hadoop to run on Windows Server and Azure environments. This incorporates porting relevant work from the similar effort on branch 1 tracked via HADOOP-8079.

Issue Links

• incorporates: YARN-176 Some YARN tests fail to find winutils
• incorporates: YARN-199 yarn cmd line scripts for windows
• incorporates: YARN-207 YARN distribution build fails on Windows
• incorporates: YARN-233 YARN support for execution of container on Windows
• incorporates: YARN-234 YARN support for Windows process tree and resource calculation
• incorporates: YARN-259 YARN and MapReduce have multiple test failures on Windows due to rejection of invalid test data path
• incorporates: YARN-316 YARN container launch may exceed maximum Windows command line length due to long classpath
• incorporates: YARN-359 NodeManager container-related tests fail on branch-trunk-win
• incorporates: HADOOP-9637 Adding Native Fstat for Windows as needed by YARN
• incorporates: YARN-390 ApplicationCLI and NodeCLI use hard-coded platform-specific line separator, which causes test failures on Windows
• incorporates: YARN-487 TestDiskFailures fails on Windows due to path mishandling
• incorporates: YARN-488 TestContainerManagerSecurity fails on Windows
• incorporates: YARN-490 TestDistributedShell fails on Windows
• incorporates: YARN-491 TestContainerLogsPage fails on Windows
• incorporates: YARN-493 NodeManager job control logic flaws on Windows
• incorporates: YARN-557 TestUnmanagedAMLauncher fails on Windows
• incorporates: YARN-593 container launch on Windows does not correctly populate classpath with new process's environment variables and localized resources
• incorporates: YARN-700 TestInfoBlock fails on Windows because of line ending missmatch
• incorporates: YARN-1025 ResourceManager and NodeManager do not load native libraries on Windows.
• incorporates: YARN-1138 yarn.application.classpath is set to point to $HADOOP_CONF_DIR etc., which does not work on Windows
• incorporates: YARN-839 TestContainerLaunch.testContainerEnvVariables fails on Windows
• incorporates: YARN-852 TestAggregatedLogFormat.testContainerLogsFileAccess fails on Windows
• incorporates: YARN-894 NodeHealthScriptRunner timeout checking is inaccurate on Windows
• incorporates: YARN-909 Disable TestLinuxContainerExecutorWithMocks on Windows
• incorporates: YARN-1077 TestContainerLaunch fails on Windows
• incorporates: YARN-1078 TestNodeManagerResync, TestNodeManagerShutdown, and TestNodeStatusUpdater fail on Windows
• incorporates: YARN-1120 Make ApplicationConstants.Environment.USER definition OS neutral
• is part of: HADOOP-8562 Enhancements to support Hadoop on Windows Server and Windows Azure environments
• relates to: YARN-456 allow OS scheduling priority of NM to be different than the containers it launches for Windows
• relates to: YARN-506 Move to common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
• relates to: YARN-597 TestFSDownload fails on Windows because of dependencies on tar/gzip/jar tools