Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8079

Proposal for enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: native
    • Labels:
    • Target Version/s:
    • Tags:
      Windows, Azure, Native Platform, Cloud, PaaS, IaaS

      Description

      This JIRA is intended to capture discussion around proposed work to enhance Apache Hadoop to run well on Windows. Apache Hadoop has worked on Microsoft Windows since its inception, but Windows support has never been a priority. Currently Windows works as a development and testing platform for Hadoop, but Hadoop is not natively integrated, full-featured or performance and scalability tuned for Windows Server or Windows Azure. We would like to change this and engage in a dialog with the broader community on the architectural design points for making Windows (enterprise and cloud) an excellent runtime and deployment environment for Hadoop.

      The Isotope team at Microsoft (names below) has developed an Apache Hadoop 1.0 patch set that addresses these performance, integration and feature gaps, allowing Apache Hadoop to be used with Azure and Windows Server without recourse to virtualization technologies such as Cygwin. We have significant interest in the deployment of Hadoop across many multi-tenant, PaaS and IaaS environments - which bring their own unique requirements.

      Microsoft has recently completed a CCLA with Apache and would like to contribute these enhancements back to the Apache Hadoop community.

      In the interest of improving Apache Hadoop so that it runs more smoothly on all platforms, including Windows, we propose first contributing this work to the Apache community by attaching it to this JIRA. From there we would like to work with the community to refine the patch set until it is ready to be merged into the Apache trunk.

      Your feedback solicited,

      Alexander Stojanovic
      Min Wei
      David Lao
      Lengning Liu
      David Zhang
      Asad Khan

        Attachments

        1. windows-cmd-scripts.patch
          27 kB
          David Lao
        2. security.patch
          6 kB
          David Lao
        3. microsoft-windowsazure-api-0.1.2.jar
          659 kB
          Min Wei
        4. mapred-tasks.patch
          25 kB
          David Lao
        5. hadoopcmdscripts.zip
          10 kB
          David Lao
        6. hadoop-8079-branch-1-win.patch
          90 kB
          David Lao
        7. hadoop-8079.patch
          26 kB
          Min Wei
        8. hadoop-8079.AzureBlobStore.patch
          0.6 kB
          Min Wei
        9. general-utils-windows.patch
          10 kB
          David Lao
        10. azurenative.zip
          11 kB
          Min Wei

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stojanovic Alexander Stojanovic
              • Votes:
                0 Vote for this issue
                Watchers:
                67 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 2,016h
                  2,016h
                  Remaining:
                  Remaining Estimate - 2,016h
                  2,016h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified