Hadoop Common
  1. Hadoop Common
  2. HADOOP-8079

Proposal for enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: native
    • Labels:
    • Target Version/s:
    • Tags:
      Windows, Azure, Native Platform, Cloud, PaaS, IaaS

      Description

      This JIRA is intended to capture discussion around proposed work to enhance Apache Hadoop to run well on Windows. Apache Hadoop has worked on Microsoft Windows since its inception, but Windows support has never been a priority. Currently Windows works as a development and testing platform for Hadoop, but Hadoop is not natively integrated, full-featured or performance and scalability tuned for Windows Server or Windows Azure. We would like to change this and engage in a dialog with the broader community on the architectural design points for making Windows (enterprise and cloud) an excellent runtime and deployment environment for Hadoop.

      The Isotope team at Microsoft (names below) has developed an Apache Hadoop 1.0 patch set that addresses these performance, integration and feature gaps, allowing Apache Hadoop to be used with Azure and Windows Server without recourse to virtualization technologies such as Cygwin. We have significant interest in the deployment of Hadoop across many multi-tenant, PaaS and IaaS environments - which bring their own unique requirements.

      Microsoft has recently completed a CCLA with Apache and would like to contribute these enhancements back to the Apache Hadoop community.

      In the interest of improving Apache Hadoop so that it runs more smoothly on all platforms, including Windows, we propose first contributing this work to the Apache community by attaching it to this JIRA. From there we would like to work with the community to refine the patch set until it is ready to be merged into the Apache trunk.

      Your feedback solicited,

      Alexander Stojanovic
      Min Wei
      David Lao
      Lengning Liu
      David Zhang
      Asad Khan

      1. windows-cmd-scripts.patch
        27 kB
        David Lao
      2. security.patch
        6 kB
        David Lao
      3. microsoft-windowsazure-api-0.1.2.jar
        659 kB
        Min Wei
      4. mapred-tasks.patch
        25 kB
        David Lao
      5. hadoopcmdscripts.zip
        10 kB
        David Lao
      6. hadoop-8079-branch-1-win.patch
        90 kB
        David Lao
      7. hadoop-8079.patch
        26 kB
        Min Wei
      8. hadoop-8079.AzureBlobStore.patch
        0.6 kB
        Min Wei
      9. general-utils-windows.patch
        10 kB
        David Lao
      10. azurenative.zip
        11 kB
        Min Wei

        Issue Links

          Activity

          Alexander Stojanovic created issue -
          Aaron T. Myers made changes -
          Field Original Value New Value
          Fix Version/s 1.0.0 [ 12318244 ]
          Target Version/s 1.0.0 [ 12318244 ] 1.1.0 [ 12316501 ]
          Aaron T. Myers made changes -
          Target Version/s 1.1.0 [ 12316501 ] 0.24.0, 1.1.0 [ 12317652, 12316501 ]
          Min Wei made changes -
          Attachment hadoop-8079.patch [ 12515381 ]
          Min Wei made changes -
          Attachment hadoop-8079.AzureBlobStore.patch [ 12515503 ]
          Attachment azurenative.zip [ 12515504 ]
          Attachment microsoft-windowsazure-api-0.1.2.jar [ 12515505 ]
          David Lao made changes -
          Attachment hadoopcmdscripts.zip [ 12515655 ]
          Sanjay Radia made changes -
          Link This issue incorporates MAPREDUCE-3898 [ MAPREDUCE-3898 ]
          David Lao made changes -
          Attachment security.patch [ 12515955 ]
          Attachment general-utils-windows.patch [ 12515956 ]
          Attachment mapred-tasks.patch [ 12515957 ]
          Attachment windows-cmd-scripts.patch [ 12515958 ]
          Doug Cutting made changes -
          Link This issue is related to HADOOP-8139 [ HADOOP-8139 ]
          David Lao made changes -
          Attachment hadoop-8079-branch-1-win.patch [ 12520233 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8101 [ HADOOP-8101 ]
          Bikas Saha made changes -
          Link This issue incorporates MAPREDUCE-4201 [ MAPREDUCE-4201 ]
          Bikas Saha made changes -
          Link This issue incorporates MAPREDUCE-4203 [ MAPREDUCE-4203 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8374 [ HADOOP-8374 ]
          Bikas Saha made changes -
          Link This issue incorporates HDFS-3424 [ HDFS-3424 ]
          Bikas Saha made changes -
          Link This issue incorporates MAPREDUCE-4260 [ MAPREDUCE-4260 ]
          Bikas Saha made changes -
          Link This issue incorporates MAPREDUCE-4263 [ MAPREDUCE-4263 ]
          Ivan Mitic made changes -
          Link This issue incorporates HADOOP-8409 [ HADOOP-8409 ]
          Ivan Mitic made changes -
          Link This issue incorporates HADOOP-8414 [ HADOOP-8414 ]
          Ivan Mitic made changes -
          Link This issue incorporates HADOOP-8411 [ HADOOP-8411 ]
          Ivan Mitic made changes -
          Link This issue incorporates HADOOP-8412 [ HADOOP-8412 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8420 [ HADOOP-8420 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8421 [ HADOOP-8421 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8424 [ HADOOP-8424 ]
          Ivan Mitic made changes -
          Link This issue incorporates HADOOP-8440 [ HADOOP-8440 ]
          Kanna Karanam made changes -
          Link This issue incorporates HADOOP-8486 [ HADOOP-8486 ]
          Ivan Mitic made changes -
          Link This issue incorporates MAPREDUCE-4321 [ MAPREDUCE-4321 ]
          Ivan Mitic made changes -
          Link This issue incorporates HADOOP-8487 [ HADOOP-8487 ]
          John Gordon made changes -
          Link This issue is related to MAPREDUCE-4322 [ MAPREDUCE-4322 ]
          Bikas Saha made changes -
          Link This issue relates to HADOOP-8492 [ HADOOP-8492 ]
          Ivan Mitic made changes -
          Link This issue incorporates HADOOP-8493 [ HADOOP-8493 ]
          Chuan Liu made changes -
          Link This issue incorporates HADOOP-8454 [ HADOOP-8454 ]
          Chuan Liu made changes -
          Link This issue incorporates HADOOP-8455 [ HADOOP-8455 ]
          Chuan Liu made changes -
          Link This issue incorporates HADOOP-8456 [ HADOOP-8456 ]
          Chuan Liu made changes -
          Link This issue incorporates HADOOP-8457 [ HADOOP-8457 ]
          Chuan Liu made changes -
          Link This issue incorporates HADOOP-8453 [ HADOOP-8453 ]
          Trupti Dhavle made changes -
          Link This issue incorporates HADOOP-8516 [ HADOOP-8516 ]
          Trupti Dhavle made changes -
          Link This issue incorporates HADOOP-8517 [ HADOOP-8517 ]
          Sumadhur Reddy Bolli made changes -
          Link This issue relates HDFS-3564 [ HDFS-3564 ]
          Bikas Saha made changes -
          Link This issue incorporates HDFS-3565 [ HDFS-3565 ]
          Sumadhur Reddy Bolli made changes -
          Link This issue relates HDFS-3566 [ HDFS-3566 ]
          Chuan Liu made changes -
          Link This issue incorporates MAPREDUCE-4374 [ MAPREDUCE-4374 ]
          Ivan Mitic made changes -
          Link This issue incorporates MAPREDUCE-4368 [ MAPREDUCE-4368 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HDFS-3564 [ HDFS-3564 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates HDFS-3564 [ HDFS-3564 ]
          Ivan Mitic made changes -
          Link This issue incorporates HADOOP-8534 [ HADOOP-8534 ]
          Trupti Dhavle made changes -
          Link This issue incorporates HADOOP-8536 [ HADOOP-8536 ]
          Trupti Dhavle made changes -
          Link This issue incorporates HADOOP-8540 [ HADOOP-8540 ]
          Ivan Mitic made changes -
          Link This issue incorporates MAPREDUCE-4386 [ MAPREDUCE-4386 ]
          Chuan Liu made changes -
          Link This issue incorporates HADOOP-8544 [ HADOOP-8544 ]
          Chuan Liu made changes -
          Link This issue incorporates HADOOP-8564 [ HADOOP-8564 ]
          Ramya Sunil made changes -
          Link This issue incorporates HADOOP-8583 [ HADOOP-8583 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8645 [ HADOOP-8645 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8420 [ HADOOP-8420 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8516 [ HADOOP-8516 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8536 [ HADOOP-8536 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8540 [ HADOOP-8540 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8564 [ HADOOP-8564 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8583 [ HADOOP-8583 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8456 [ HADOOP-8456 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8517 [ HADOOP-8517 ]
          Bikas Saha made changes -
          Link This issue incorporates MAPREDUCE-4386 [ MAPREDUCE-4386 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8455 [ HADOOP-8455 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8487 [ HADOOP-8487 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8457 [ HADOOP-8457 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8493 [ HADOOP-8493 ]
          Bikas Saha made changes -
          Link This issue incorporates MAPREDUCE-4374 [ MAPREDUCE-4374 ]
          Bikas Saha made changes -
          Link This issue incorporates HADOOP-8453 [ HADOOP-8453 ]
          Enis Soztutar made changes -
          Link This issue is related to HBASE-6814 [ HBASE-6814 ]
          Slavik Krassovsky made changes -
          Link This issue relates to HADOOP-8900 [ HADOOP-8900 ]
          Matt Foley made changes -
          Target Version/s 1.1.0, 0.24.0 [ 12316501, 12317652 ] 1.2.0, 0.24.0 [ 12321659, 12317652 ]
          Ganeshan Iyer made changes -
          Link This issue incorporates HADOOP-9062 [ HADOOP-9062 ]
          Ivan Mitic made changes -
          Link This issue incorporates HADOOP-9062 [ HADOOP-9062 ]
          Gavin made changes -
          Link This issue relates to HDFS-3566 [ HDFS-3566 ]
          Gavin made changes -
          Link This issue relates to HDFS-3566 [ HDFS-3566 ]
          Matt Foley made changes -
          Target Version/s 1.2.0, 0.24.0 [ 12321659, 12317652 ] 0.24.0, 1.3.0 [ 12317652, 12324327 ]
          Chris Nauroth made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Alexander Stojanovic
            • Votes:
              0 Vote for this issue
              Watchers:
              67 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 2,016h
                2,016h
                Remaining:
                Remaining Estimate - 2,016h
                2,016h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development