Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8562

Enhancements to support Hadoop on Windows Server and Windows Azure environments

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0-alpha1
    • 2.1.0-beta
    • None
    • None
    • Reviewed
    • This umbrella jira makes enhancements to support Hadoop natively on Windows Server and Windows Azure environments.
    • Windows

    Description

      This JIRA tracks the work that needs to be done on trunk to enable Hadoop to run on Windows Server and Azure environments. This incorporates porting relevant work from the similar effort on branch 1 tracked via HADOOP-8079.

      Attachments

        1. test-untar.tgz
          2 kB
          Chris Nauroth
        2. test-untar.tar
          20 kB
          Chris Nauroth
        3. branch-trunk-win-min.patch
          60 kB
          Suresh Srinivas
        4. branch-trunk-win.patch
          738 kB
          Suresh Srinivas
        5. branch-trunk-win.patch
          609 kB
          Suresh Srinivas
        6. branch-trunk-win.patch
          635 kB
          Suresh Srinivas
        7. branch-trunk-win.patch
          544 kB
          Suresh Srinivas
        8. branch-trunk-win.patch
          557 kB
          Suresh Srinivas
        9. branch-trunk-win.patch
          543 kB
          Suresh Srinivas
        10. branch-trunk-win.patch
          565 kB
          Suresh Srinivas
        11. branch-trunk-win.patch
          561 kB
          Suresh Srinivas
        12. branch-trunk-win.patch
          593 kB
          Suresh Srinivas
        13. branch-trunk-win.patch
          593 kB
          Suresh Srinivas
        14. branch-trunk-win.patch
          642 kB
          Suresh Srinivas
        15. branch-trunk-win.min-notest.patch
          225 kB
          Suresh Srinivas
        16. branch-trunk-win.min.patch
          340 kB
          Suresh Srinivas
        17. branch-2.merge.patch
          640 kB
          Arpit Agarwal
        18. branch-2.merge.patch
          640 kB
          Arpit Agarwal

        Issue Links

        breaks

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9397 Incremental dist tar build fails

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9481 Broken conditional logic with HADOOP_SNAPPY_LIBRARY

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed
        incorporates

        Sub-task - The sub-task of the issue HADOOP-8980 TestRPC and TestSaslRPC fail on Windows

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9133 Windows tarball build fails when enlistment root path is long

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8953 Shell PathData parsing failures on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8954 "stat" executable not found on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8955 "chmod" executable not found on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8956 FileSystem.primitiveMkdir failures on Windows cause multiple test suites to fail

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8959 TestUserGroupInformation fails on Windows due to "id" executable not found

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8960 TestMetricsServlet fails on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8961 GenericOptionsParser URI parsing failure on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8974 TestDFVariations fails on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8975 TestFileContextResolveAfs fails on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8976 TestFSMainOperationsLocalFileSystem fails on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8977 multiple FsShell test failures on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8978 TestTrash fails on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8979 TestHttpServer fails on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9005 Merge hadoop cmd line scripts from branch-1-win

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9008 Building hadoop tarball fails on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9011 saveVersion.py does not include branch in version annotation

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9110 winutils ls off-by-one error indexing MONTHS array can cause access violation

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9144 FindBugs reports new warnings in branch-trunk-win

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9146 failure to set sticky bit regression on branch-trunk-win

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9266 correct javac, findbugs, and release audit warnings on branch-trunk-win

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9309 test failures on Windows due to UnsatisfiedLinkError in NativeCodeLoader#buildSupportsSnappy

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9313 Remove spurious mkdir from hadoop-config.cmd

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9368 Add timeouts to new tests in branch-trunk-win

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HDFS-4241 DataNode unit test failures on Windows

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9687 branch-1-win TestJobTrackerQuiescence and TestFileLengthOnClusterRestart failed caused by incorrect DFS path construction on Windows

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9759 Add support for NativeCodeLoader#getLibraryName on Windows

        • Critical - Crashes, loss of data, severe memory leak.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8948 TestFileUtil.testGetDU fails on Windows due to incorrect assumption of line separator

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8957 AbstractFileSystem#IsValidName should be overridden for embedded file systems like ViewFs

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8958 ViewFs:Non absolute mount name failures when running multiple tests on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8973 DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8981 TestMetricsSystemImpl fails on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8982 TestSocketIOWithTimeout fails on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9043 disallow in winutils creating symlinks with forwards slashes

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9131 TestLocalFileSystem#testListStatusWithColons cannot run on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9264 port change to use Java untar API on Windows from branch-1-win to trunk

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9278 HarFileSystem may leak file handle

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9290 Some tests cannot load native library

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9294 GetGroupsTestBase fails on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9364 PathData#expandAsGlob does not return correct results for absolute paths on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9365 TestHAZKUtil fails on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9372 Fix bad timeout annotations on tests

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9437 TestNativeIO#testRenameTo fails on Windows due to assumption that POSIX errno is embedded in NativeIOException

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9443 Port winutils static code analysis change to trunk

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9488 FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9500 TestUserGroupInformation#testGetServerSideGroups fails on Windows due to failure to find winutils.exe

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9524 Fix ShellCommandFencer to work on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9526 TestShellCommandFencer and TestShell fail on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9527 Add symlink support to LocalFileSystem on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9553 TestAuthenticationToken fails on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-10030 FsShell -put/copyFromLocal should support Windows local path

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9656 Gridmix unit tests fail on Windows and Linux

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9681 FileUtil.unTarUsingJava() should close the InputStream upon finishing

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9422 HADOOP_HOME should not be required to be set to be able to launch commands using hadoop.util.Shell

        • Major - Major loss of function.
        • Patch Available

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9018 Reject invalid Windows URIs

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9056 Build native library on Windows

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9081 Port TestWinUtils from branch-1-win to branch-trunk-win

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9347 add instructions to BUILDING.txt describing how to build on Windows

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9356 remove remaining references to cygwin/cygpath from scripts

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HDFS-3602 Enhancements to HDFS for Windows Server and Windows Azure development and runtime environments

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4401 Enhancements to MapReduce for Windows Server and Windows Azure development and runtime environments

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. YARN-191 Enhancements to YARN for Windows Server and Windows Azure development and runtime environments

        • Major - Major loss of function.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9348 address TODO in winutils to add more command line usage and examples

        • Trivial - Cosmetic problem like misspelt words or misaligned text.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9354 Windows native project files missing license headers

        • Trivial - Cosmetic problem like misspelt words or misaligned text.
        • Resolved

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-9483 winutils support for readlink command

        • Major - Major loss of function.
        • Closed

        Test - A new unit, integration or system test. HDFS-4554 TestLocalDirAllocator and TestDiskError need to be reviewed for Windows compatibility and either re-enabled or re-documented.

        • Trivial - Cosmetic problem like misspelt words or misaligned text.
        • Open

        Task - A task that needs to be done. HADOOP-8945 winutils: merge codebase from branch-1-win to branch-trunk-win

        • Major - Major loss of function.
        • Resolved

        Task - A task that needs to be done. HADOOP-8946 winutils: compile codebase during Maven build on branch-trunk-win

        • Major - Major loss of function.
        • Resolved

        Task - A task that needs to be done. HADOOP-8947 Merge FileUtil and Shell changes from branch-1-win to branch-trunk-win to enable initial test pass

        • Major - Major loss of function.
        • Resolved

        Task - A task that needs to be done. HADOOP-8949 FileUtil.CygPathCommand is dead code on branch-1-win and branch-trunk-win

        • Major - Major loss of function.
        • Resolved
        is related to

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9600 In Windows: Hadoop fails to run when JAVA_HOME has spaces in it

        • Major - Major loss of function.
        • Open

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9433 TestLocalFileSystem#testHasFileDescriptor leaks file handle

        • Major - Major loss of function.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9431 TestSecurityUtil#testLocalHostNameForNullOrWild on systems where hostname contains capital letters

        • Trivial - Cosmetic problem like misspelt words or misaligned text.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9555 HA functionality that uses ZooKeeper may experience inadvertent TCP RST and miss session expiration event due to bug in client connection management

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9556 disable HA tests on Windows that fail due to ZooKeeper client connection management bug

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9632 TestShellCommandFencer will fail if there is a 'host' machine in the network

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed
        relates to

        Sub-task - The sub-task of the issue HADOOP-16045 Don't run TestDU on Windows

        • Trivial - Cosmetic problem like misspelt words or misaligned text.
        • Resolved

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9376 TestProxyUserFromEnv fails on a Windows domain joined machine

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9388 TestFsShellCopy fails on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9413 Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9490 LocalFileSystem#reportChecksumFailure not closing the checksum file handle before rename

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9599 hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9678 TestRPC#testStopsAllThreads intermittently fails on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HDFS-4572 Fix TestJournal failures on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HDFS-4584 Fix TestNNWithQJM failures on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HDFS-4615 Fix TestDFSShell failures on Windows

        • Major - Major loss of function.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9373 Merge CHANGES.branch-trunk-win.txt to CHANGES.txt

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-9387 TestDFVariations fails on Windows after the merge

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Closed

        Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14309 Add PowerShell NodeFencer

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Test - A new unit, integration or system test. HADOOP-9525 Add tests that validate winutils chmod behavior on folders

        • Major - Major loss of function.
        • Closed
        supercedes

        Bug - A problem which impairs or prevents the functions of the product. HADOOP-8621 FileUtil.symLink fails if spaces in path

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            bikassaha Bikas Saha
            bikassaha Bikas Saha
            Votes:
            0 Vote for this issue
            Watchers:
            22 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment