[HADOOP-8645] Stabilize branch-1-win

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Done
• Assignee: Unassigned
• Reporter: Bikas Saha
• Created: 03/Aug/12 05:25
• Updated: 13/Feb/15 21:20
• Resolved: 13/Feb/15 21:20

Description

Most of the code changes to make Hadoop branch 1 work natively on Windows are done. This jira in intended to track the work needed to achieve 100% test pass for the dev tests.

Issue Links

• blocks: HADOOP-10206 Port WASB HBase support to Hadoop 2.0
• incorporates: HADOOP-8874 HADOOP_HOME and -Dhadoop.home (from hadoop wrapper script) are not uniformly handled
• incorporates: MAPREDUCE-4386 Investigate possible abstractions for shell commands "bash -c" and "cmd /c"
• incorporates: HADOOP-8455 Address user name format on domain joined Windows machines
• incorporates: HADOOP-8493 Extend Path with Path#toFile() and Path(File) to better support path cross-platform differences
• incorporates: HADOOP-8420 Hadoop Common creating package-info.java must not depend on sh
• incorporates: HADOOP-8487 Many HDFS tests use a test path intended for local file system tests
• incorporates: HADOOP-8516 fsck command does not work when executed on Windows Hadoop installation
• incorporates: HADOOP-8517 --config option does not work with Hadoop installation on Windows
• incorporates: HADOOP-8536 Problem with -Dproperty=value option on windows hadoop
• incorporates: HADOOP-8540 Compression with non-default codec's fail on Windows
• incorporates: HADOOP-8564 Port and extend Hadoop native libraries for Windows to address datanode concurrent reading and writing issue
• incorporates: HADOOP-8583 Globbing is not correctly handled in a few cases on Windows
• incorporates: HADOOP-8657 TestCLI fails on Windows because it uses hardcoded file length of test files committed to the source code
• incorporates: HADOOP-8664 hadoop streaming job need the full path to commands even when they are in the path
• incorporates: HADOOP-8694 Create true symbolic links on Windows
• incorporates: HADOOP-8731 TestTrackerDistributedCacheManager fails on Windows
• incorporates: HADOOP-8732 Address intermittent test failures on Windows
• incorporates: HADOOP-8733 TestStreamingTaskLog, TestJvmManager, TestLinuxTaskControllerLaunchArgs fail on Windows
• incorporates: HADOOP-8734 LocalJobRunner does not support private distributed cache
• incorporates: HADOOP-8739 Cmd scripts for Windows have issues in argument parsing
• incorporates: HADOOP-8868 FileUtil#chmod should normalize the path before calling into shell APIs
• incorporates: HADOOP-8869 Links at the bottom of the jobdetails page do not render correctly in IE9
• incorporates: HADOOP-8872 FileSystem#length returns zero for symlinks on windows+java6
• incorporates: HADOOP-8899 Classpath exceeds maximum OS limit
• incorporates: HADOOP-8902 Enable Gridmix v1 & v2 benchmarks on Windows platform
• incorporates: HADOOP-8903 support HADOOP_USER_CLASSPATH_FIRST in windows hadoop cmd
• incorporates: HADOOP-8907 Provide means to look for zlib1.dll next to hadoop.dll on Windows
• incorporates: HADOOP-8972 Move winutils tests from bat to Java
• incorporates: HADOOP-9026 Hadoop.cmd fails to initialize if user's %path% variable has parenthesis
• incorporates: HADOOP-9027 Build fails on Windows without sh/sed/echo in the path
• incorporates: HADOOP-9057 TestMetricsSystemImpl.testInitFirst fails intermittently
• incorporates: HADOOP-9061 Java6+Windows does not work well with symlinks
• incorporates: HADOOP-9062 hadoop-env.cmd overwrites the value of *_OPTS set before install
• incorporates: HADOOP-9074 Hadoop install scripts for Windows
• incorporates: HADOOP-9096 Improve performance of Windows install scripts
• incorporates: HADOOP-9177 Address issues that came out from running static code analysis on winutils
• incorporates: HADOOP-9250 Windows installer bugfixes
• incorporates: HADOOP-9452 Windows install scripts bugfixes
• incorporates: HADOOP-9463 branch-1-win fails to build with OpenJDK7
• incorporates: HADOOP-9480 The windows installer should pick the config from src\packages\win\template\conf
• incorporates: HADOOP-9551 Backport common utils introduced with HADOOP-9413 to branch-1-win
• incorporates: HADOOP-9552 Windows log4j template should suppress info messages from mortbay.log
• incorporates: HADOOP-9579 Contrib ant test target not setting the java.library.path
• incorporates: HADOOP-9677 TestSetupAndCleanupFailure#testWithDFS fails on Windows
• incorporates: HDFS-3763 TestNameNodeMXBean fails on Windows
• incorporates: HDFS-3766 Fix TestStorageRestore on Windows
• incorporates: MAPREDUCE-4510 Avoid logging "Cannot run program getconf" on Windows
• incorporates: MAPREDUCE-4561 Support for node health scripts on Windows
• incorporates: MAPREDUCE-4564 Shell timeout mechanism does not work for processes spawned using winutils
• incorporates: MAPREDUCE-4597 TestKillSubProcesses intermittently fails
• incorporates: MAPREDUCE-4598 Support for node health scripts on Windows
• incorporates: MAPREDUCE-5277 Job history completed location cannot be on a file system other than default
• incorporates: HADOOP-8456 Support spaces in user names and group names in results returned via winutils
• incorporates: HADOOP-8457 Address file ownership issue for users in Administrators group on Windows.
• incorporates: HADOOP-8763 Set group owner on Windows failed
• incorporates: HADOOP-8836 UGI should throw exception in case winutils.exe cannot be loaded
• incorporates: HADOOP-8879 TestUserGroupInformation fails on Windows when runas Administrator
• incorporates: HADOOP-8935 Make 'winutils ls' show the SID if the owner does not exist on the system
• incorporates: HADOOP-8936 Local users should be able to query a domain user's groups on Windows
• incorporates: HADOOP-9006 Winutils should keep Administrators privileges intact
• incorporates: HADOOP-9007 TestJobStatusPersistency.testJobStoreDisablingWithInvalidPath failure
• incorporates: HADOOP-9472 Cleanup hadoop-config.cmd
• incorporates: HADOOP-9609 Remove sh dependency of bin-package target
• incorporates: HDFS-4341 TestBalancerWithNodeGroup fails on Windows
• incorporates: MAPREDUCE-4657 WindowsResourceCalculatorPlugin has NPE
• incorporates: MAPREDUCE-5224 JobTracker should allow the system directory to be in non-default FS
• incorporates: HADOOP-9036 TestSinkQueue.testConcurrentConsumers fails intermittently (Backports HADOOP-7292)
• incorporates: HADOOP-9507 LocalFileSystem rename() is broken in some cases when destination exists
• incorporates: MAPREDUCE-4374 Fix child task environment variable config and add support for Windows
• incorporates: MAPREDUCE-5387 Implement Signal.TERM on Windows
• incorporates: HADOOP-8847 Change untar to use Java API on Windows instead of spawning tar process
• incorporates: HADOOP-8908 Refactor 'winutils.exe' code
• incorporates: HADOOP-8601 Extend TestShell to cover Windows shell commands
• incorporates: HADOOP-9624 TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has "X" in its name
• incorporates: HADOOP-8453 Add unit tests for winutils
• is part of: HADOOP-8079 Proposal for enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments
• is related to: HADOOP-8900 BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match decompressed size
• is related to: HDFS-3833 TestDFSShell fails on Windows due to file concurrent read write
• is related to: HBASE-6814 [WINDOWS] HBase on Windows
• relates to: HADOOP-8900 BuiltInGzipDecompressor throws IOException - stored gzip size doesn't match decompressed size
• relates to: HDFS-4677 Editlog should support synchronous writes

Sub-Tasks

• HADOOP_HOME and -Dhadoop.home (from hadoop wrapper script) are not uniformly handled: Resolved