Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8468

Umbrella of enhancements to support different failure and locality topologies

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0, 2.0.0-alpha
    • Fix Version/s: None
    • Component/s: ha, io
    • Labels:
      None

      Description

      The current hadoop network topology (described in some previous issues like: Hadoop-692) works well in classic three-tiers network when it comes out. However, it does not take into account other failure models or changes in the infrastructure that can affect network bandwidth efficiency like: virtualization.
      Virtualized platform has following genes that shouldn't been ignored by hadoop topology in scheduling tasks, placing replica, do balancing or fetching block for reading:
      1. VMs on the same physical host are affected by the same hardware failure. In order to match the reliability of a physical deployment, replication of data across two virtual machines on the same host should be avoided.
      2. The network between VMs on the same physical host has higher throughput and lower latency and does not consume any physical switch bandwidth.
      Thus, we propose to make hadoop network topology extend-able and introduce a new level in the hierarchical topology, a node group level, which maps well onto an infrastructure that is based on a virtualized environment.

        Attachments

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                djp Junping Du
                Reporter:
                djp Junping Du
              • Votes:
                6 Vote for this issue
                Watchers:
                65 Start watching this issue

                Dates

                • Created:
                  Updated: