Hadoop Common
  1. Hadoop Common
  2. HADOOP-8468

Umbrella of enhancements to support different failure and locality topologies

    Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0, 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: ha, io
    • Labels:
      None

      Description

      The current hadoop network topology (described in some previous issues like: Hadoop-692) works well in classic three-tiers network when it comes out. However, it does not take into account other failure models or changes in the infrastructure that can affect network bandwidth efficiency like: virtualization.
      Virtualized platform has following genes that shouldn't been ignored by hadoop topology in scheduling tasks, placing replica, do balancing or fetching block for reading:
      1. VMs on the same physical host are affected by the same hardware failure. In order to match the reliability of a physical deployment, replication of data across two virtual machines on the same host should be avoided.
      2. The network between VMs on the same physical host has higher throughput and lower latency and does not consume any physical switch bandwidth.
      Thus, we propose to make hadoop network topology extend-able and introduce a new level in the hierarchical topology, a node group level, which maps well onto an infrastructure that is based on a virtualized environment.

      1. HVE User Guide on branch-1(draft ).pdf
        380 kB
        Junping Du
      2. HVE_Hadoop World Meetup 2012.pptx
        954 kB
        Junping Du
      3. Proposal for enchanced failure and locality topologies (revised-1.0).pdf
        269 kB
        Junping Du
      4. HADOOP-8468-total-v3.patch
        259 kB
        Junping Du
      5. HADOOP-8468-total.patch
        259 kB
        Junping Du
      6. Proposal for enchanced failure and locality topologies.pdf
        260 kB
        Junping Du

        Issue Links

          Activity

          Junping Du created issue -
          Junping Du made changes -
          Field Original Value New Value
          Attachment Proposal for enchanced failure and locality topologies.pdf [ 12530720 ]
          Junping Du made changes -
          Attachment HADOOP-8468-total.patch [ 12530721 ]
          Junping Du made changes -
          Attachment HADOOP-8468-total-v3.patch [ 12530786 ]
          Junping Du made changes -
          Link This issue incorporates HDFS-3495 [ HDFS-3495 ]
          Junping Du made changes -
          Link This issue incorporates MAPREDUCE-4309 [ MAPREDUCE-4309 ]
          Junping Du made changes -
          Link This issue incorporates MAPREDUCE-4310 [ MAPREDUCE-4310 ]
          Junping Du made changes -
          Link This issue incorporates HDFS-3498 [ HDFS-3498 ]
          Junping Du made changes -
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HDFS-3564 [ HDFS-3564 ]
          Tsz Wo Nicholas Sze made changes -
          Link This issue incorporates HDFS-3601 [ HDFS-3601 ]
          Suresh Srinivas made changes -
          Issue Type Bug [ 1 ] Improvement [ 4 ]
          Priority Critical [ 2 ] Major [ 3 ]
          Junping Du made changes -
          Target Version/s 2.2.0-alpha [ 12322473 ]
          Junping Du made changes -
          Attachment HVE_Hadoop World Meetup 2012.pptx [ 12550625 ]
          Junping Du made changes -
          Attachment HVE User Guide on branch-1(draft ).pdf [ 12551386 ]
          Junping Du made changes -
          Link This issue is related to HADOOP-9045 [ HADOOP-9045 ]
          Junping Du made changes -
          Link This issue relates to HADOOP-9045 [ HADOOP-9045 ]
          Junping Du made changes -
          Link This issue relates to HADOOP-9045 [ HADOOP-9045 ]
          Junping Du made changes -
          Link This issue relates to HADOOP-9045 [ HADOOP-9045 ]
          Junping Du made changes -
          Link This issue relates to HADOOP-9045 [ HADOOP-9045 ]
          Junping Du made changes -
          Link This issue is related to HADOOP-9045 [ HADOOP-9045 ]
          Junping Du made changes -
          Link This issue incorporates HADOOP-9045 [ HADOOP-9045 ]
          Jan Kunigk made changes -
          Component/s ha [ 12316608 ]
          Jan Kunigk made changes -
          Component/s ha [ 12316608 ]
          Junping Du made changes -
          Link This issue is related to HDFS-4886 [ HDFS-4886 ]
          Luke Lu made changes -
          Fix Version/s 1.2.0 [ 12321659 ]
          Target Version/s 2.0.2-alpha [ 12322473 ] 2.1.0-beta [ 12324030 ]
          Junping Du made changes -
          Link This issue is related to HDFS-4898 [ HDFS-4898 ]
          Arpit Agarwal made changes -
          Fix Version/s 2.1.0-beta [ 12324030 ]
          Junping Du made changes -
          Issue Type Improvement [ 4 ] New Feature [ 2 ]
          Allen Wittenauer made changes -
          Fix Version/s 1.2.0 [ 12321659 ]
          Fix Version/s 2.1.0-beta [ 12324030 ]
          Allen Wittenauer made changes -
          Link This issue relates to HADOOP-11326 [ HADOOP-11326 ]

            People

            • Assignee:
              Junping Du
              Reporter:
              Junping Du
            • Votes:
              6 Vote for this issue
              Watchers:
              62 Start watching this issue

              Dates

              • Created:
                Updated:

                Development