Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.2.0
    • Component/s: None
    • Labels:
    • Target Version/s:
    • Release Note:
      A new 4-layer network topology NetworkToplogyWithNodeGroup is available to make Hadoop more robust and efficient in virtualized environment.

      Description

      HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes:
      1. Add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470
      2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601
      3. Update balancer policy:HDFS-3495
      4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality: YARN-18, YARN-19.
      This JIRA will address the backport work on branch-1 which will be divided into 4 issues/patches in related jira issues.

      1. HADOOP-8817.patch
        156 kB
        Junping Du
      2. HADOOP-8817-v2.patch
        172 kB
        Junping Du
      3. HADOOP-8817-v3.patch
        172 kB
        Junping Du
      4. HADOOP-8817-v4.patch
        170 kB
        Junping Du

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          184d 2h 12m 1 Luke Lu 19/Mar/13 17:41
          Resolved Resolved Closed Closed
          56d 11h 34m 1 Matt Foley 15/May/13 05:16
          Junping Du made changes -
          Link This issue is duplicated by HADOOP-12283 [ HADOOP-12283 ]
          RacingDawn made changes -
          Link This issue is cloned by HADOOP-12283 [ HADOOP-12283 ]
          RacingDawn made changes -
          Link This issue is cloned by HADOOP-12283 [ HADOOP-12283 ]
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Luke Lu made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Luke Lu added a comment -

          This jira is covered by JIRAs it "contains".

          Show
          Luke Lu added a comment - This jira is covered by JIRAs it "contains".
          Luke Lu made changes -
          Release Note A new 4-layer network topology NetworkToplogyWithNodeGroup is available to make Hadoop more robust and efficient in virtualized environment.
          Fix Version/s 1.2.0 [ 12321659 ]
          Junping Du made changes -
          Attachment HADOOP-8817-v4.patch [ 12553300 ]
          Hide
          Junping Du added a comment -

          Address Jing's comments with addressing one NPE issue and remove some redundant code.

          Show
          Junping Du added a comment - Address Jing's comments with addressing one NPE issue and remove some redundant code.
          Hide
          Junping Du added a comment -

          I meet some issues in running ant test on local environment that even branch-1 without patch cannot pass all tests. I run related unit tests which is good and did lots of tests on real cluster for this patch (reliability and performance) which you can find in attachment in HADOOP-8848. Anyone can help to verify this patch to pass "ant test"?

          Show
          Junping Du added a comment - I meet some issues in running ant test on local environment that even branch-1 without patch cannot pass all tests. I run related unit tests which is good and did lots of tests on real cluster for this patch (reliability and performance) which you can find in attachment in HADOOP-8848 . Anyone can help to verify this patch to pass "ant test"?
          Junping Du made changes -
          Attachment HADOOP-8817-v3.patch [ 12553250 ]
          Hide
          Junping Du added a comment -

          Update patch slightly due to Nicholas's comments in HADOOP-8820

          Show
          Junping Du added a comment - Update patch slightly due to Nicholas's comments in HADOOP-8820
          Hide
          Junping Du added a comment -

          Luke. Thanks for the review and comments. I will attach these results soon.

          Show
          Junping Du added a comment - Luke. Thanks for the review and comments. I will attach these results soon.
          Hide
          Luke Lu added a comment -

          The changes lgtm. Please post ant test and test-patch results. Thanks Junping.

          Show
          Luke Lu added a comment - The changes lgtm. Please post ant test and test-patch results. Thanks Junping.
          Junping Du made changes -
          Attachment HADOOP-8817-v2.patch [ 12551892 ]
          Hide
          Junping Du added a comment -

          Updated patch with rebase to recent commits on branch-1 and bug fix.

          Show
          Junping Du added a comment - Updated patch with rebase to recent commits on branch-1 and bug fix.
          Hide
          Junping Du added a comment -

          Thanks Suresh for the correction here. So branch-1.2 is next feature branch (1.1 is next bug-fix branch). Isn't it?

          Show
          Junping Du added a comment - Thanks Suresh for the correction here. So branch-1.2 is next feature branch (1.1 is next bug-fix branch). Isn't it?
          Junping Du made changes -
          Link This issue contains MAPREDUCE-4660 [ MAPREDUCE-4660 ]
          Junping Du made changes -
          Link This issue relates to MAPREDUCE-4660 [ MAPREDUCE-4660 ]
          Junping Du made changes -
          Link This issue contains HDFS-3942 [ HDFS-3942 ]
          Junping Du made changes -
          Link This issue relates to HDFS-3942 [ HDFS-3942 ]
          Junping Du made changes -
          Link This issue contains HDFS-3941 [ HDFS-3941 ]
          Junping Du made changes -
          Link This issue relates to HDFS-3941 [ HDFS-3941 ]
          Junping Du made changes -
          Link This issue contains HADOOP-8820 [ HADOOP-8820 ]
          Junping Du made changes -
          Link This issue relates to HADOOP-8820 [ HADOOP-8820 ]
          Suresh Srinivas made changes -
          Target Version/s 1.0.3 [ 12320248 ] 1.2.0 [ 12321659 ]
          Hide
          Suresh Srinivas added a comment -

          Target version cannot be 1.0.3 since it is already released.

          Show
          Suresh Srinivas added a comment - Target version cannot be 1.0.3 since it is already released.
          Junping Du made changes -
          Link This issue relates to MAPREDUCE-4660 [ MAPREDUCE-4660 ]
          Junping Du made changes -
          Link This issue relates to HDFS-3942 [ HDFS-3942 ]
          Junping Du made changes -
          Link This issue relates to HDFS-3941 [ HDFS-3941 ]
          Junping Du made changes -
          Link This issue relates to HADOOP-8820 [ HADOOP-8820 ]
          Junping Du made changes -
          Attachment HADOOP-8817.patch [ 12545325 ]
          Hide
          Junping Du added a comment -

          Attach the whole patch as an example and will divided into 4 patches later.

          Show
          Junping Du added a comment - Attach the whole patch as an example and will divided into 4 patches later.
          Junping Du made changes -
          Description HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes:
          1. Add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470
          2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601
          3. Update balancer policy:HDFS-3495
          4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality: YARN-18, YARN-19.
          This JIRA will address the backport work which will be divided into 4 issues/patches in related jiras.
          HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes:
          1. Add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470
          2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601
          3. Update balancer policy:HDFS-3495
          4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality: YARN-18, YARN-19.
          This JIRA will address the backport work on branch-1 which will be divided into 4 issues/patches in related jira issues.
          Junping Du made changes -
          Description HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes:
          1. Add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470
          2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601
          3. Update balancer policy:HDFS-3495
          4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality.
          HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes:
          1. Add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470
          2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601
          3. Update balancer policy:HDFS-3495
          4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality: YARN-18, YARN-19.
          This JIRA will address the backport work which will be divided into 4 issues/patches in related jiras.
          Junping Du made changes -
          Field Original Value New Value
          Labels features
          Target Version/s 1.0.3 [ 12320248 ]
          Description HADOOP-8468 propose network topology changes for running on virtualized infrastructure, which includes:
          1. Add "NodeGroup" layer in new NetworkTopology (also known as NetworkTopologyWithNodeGroup): HADOOP-8469, HADOOP-8470
          2. Update Replica Placement/Removal Policy to reflect new topology layer: HDFS-3498, HDFS-3601
          3. Update balancer policy:HDFS-3495
          4. Update Task Scheduling Policy to reflect new topology layer and support the case that compute nodes (NodeManager or TaskTracker) and data nodes are separated into different VMs, but still benefit from physical host locality.
          Junping Du created issue -

            People

            • Assignee:
              Junping Du
              Reporter:
              Junping Du
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development