Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
    • Target Version/s:
    • Release Note:
      A new 4-layer network topology NetworkToplogyWithNodeGroup is available to make Hadoop more robust and efficient in virtualized environment.

      Description

      HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes:
      1. Add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470
      2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601
      3. Update balancer policy:HDFS-3495
      4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality: YARN-18, YARN-19.
      This JIRA will address the backport work on branch-1 which will be divided into 4 issues/patches in related jira issues.

      1. HADOOP-8817-v4.patch
        170 kB
        Junping Du
      2. HADOOP-8817-v3.patch
        172 kB
        Junping Du
      3. HADOOP-8817-v2.patch
        172 kB
        Junping Du
      4. HADOOP-8817.patch
        156 kB
        Junping Du

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Hide
          Luke Lu added a comment -

          This jira is covered by JIRAs it "contains".

          Show
          Luke Lu added a comment - This jira is covered by JIRAs it "contains".
          Hide
          Junping Du added a comment -

          Address Jing's comments with addressing one NPE issue and remove some redundant code.

          Show
          Junping Du added a comment - Address Jing's comments with addressing one NPE issue and remove some redundant code.
          Hide
          Junping Du added a comment -

          I meet some issues in running ant test on local environment that even branch-1 without patch cannot pass all tests. I run related unit tests which is good and did lots of tests on real cluster for this patch (reliability and performance) which you can find in attachment in HADOOP-8848. Anyone can help to verify this patch to pass "ant test"?

          Show
          Junping Du added a comment - I meet some issues in running ant test on local environment that even branch-1 without patch cannot pass all tests. I run related unit tests which is good and did lots of tests on real cluster for this patch (reliability and performance) which you can find in attachment in HADOOP-8848 . Anyone can help to verify this patch to pass "ant test"?
          Hide
          Junping Du added a comment -

          Update patch slightly due to Nicholas's comments in HADOOP-8820

          Show
          Junping Du added a comment - Update patch slightly due to Nicholas's comments in HADOOP-8820
          Hide
          Junping Du added a comment -

          Luke. Thanks for the review and comments. I will attach these results soon.

          Show
          Junping Du added a comment - Luke. Thanks for the review and comments. I will attach these results soon.
          Hide
          Luke Lu added a comment -

          The changes lgtm. Please post ant test and test-patch results. Thanks Junping.

          Show
          Luke Lu added a comment - The changes lgtm. Please post ant test and test-patch results. Thanks Junping.
          Hide
          Junping Du added a comment -

          Updated patch with rebase to recent commits on branch-1 and bug fix.

          Show
          Junping Du added a comment - Updated patch with rebase to recent commits on branch-1 and bug fix.
          Hide
          Junping Du added a comment -

          Thanks Suresh for the correction here. So branch-1.2 is next feature branch (1.1 is next bug-fix branch). Isn't it?

          Show
          Junping Du added a comment - Thanks Suresh for the correction here. So branch-1.2 is next feature branch (1.1 is next bug-fix branch). Isn't it?
          Hide
          Suresh Srinivas added a comment -

          Target version cannot be 1.0.3 since it is already released.

          Show
          Suresh Srinivas added a comment - Target version cannot be 1.0.3 since it is already released.
          Hide
          Junping Du added a comment -

          Attach the whole patch as an example and will divided into 4 patches later.

          Show
          Junping Du added a comment - Attach the whole patch as an example and will divided into 4 patches later.

            People

            • Assignee:
              Junping Du
              Reporter:
              Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development