Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-2681

Support bandwidth enforcement for containers while reading from HDFS

    Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.5.1
    • Fix Version/s: None
    • Component/s: nodemanager
    • Labels:
      None
    • Environment:

      Linux

    • Target Version/s:
    • Release Note:
      support HDFS read bandwidth enforcement in case data is tranfered over TCP/IP socket

      Description

      To read/write data from HDFS on data node, applications establise TCP/IP connections with the datanode. The HDFS read can be controled by setting Linux Traffic Control (TC) subsystem on the data node to make filters on appropriate connections.

      The current cgroups net_cls concept can not be applied on the node where the container is launched, netheir on data node since:

      • TC hanldes outgoing bandwidth only, so it can be set on container node (HDFS read = incoming data for the container)
      • Since HDFS data node is handled by only one process, it is not possible to use net_cls to separate connections from different containers to the datanode.

      Tasks:
      1) Extend Resource model to define bandwidth enforcement rate
      2) Monitor TCP/IP connection estabilised by container handling process and its child processes
      3) Set Linux Traffic Control rules on data node base on address:port pairs in order to enforce bandwidth of outgoing data

      Concept: http://www.hit.bme.hu/~do/papers/EnforcementDesign.pdf
      Implementation:
      http://www.hit.bme.hu/~dohoai/documents/HdfsTrafficControl.pdf
      http://www.hit.bme.hu/~dohoai/documents/HdfsTrafficControl_UML_diagram.png

        Attachments

        1. YARN-2681.patch
          340 kB
          Nam H. Do
        2. YARN-2681.005.patch
          363 kB
          Nam H. Do
        3. YARN-2681.004.patch
          363 kB
          Nam H. Do
        4. YARN-2681.003.patch
          352 kB
          Nam H. Do
        5. YARN-2681.002.patch
          348 kB
          Nam H. Do
        6. YARN-2681.001.patch
          348 kB
          Nam H. Do
        7. Traffic Control Design.png
          147 kB
          Nam H. Do

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                cntic Nam H. Do
              • Votes:
                0 Vote for this issue
                Watchers:
                24 Start watching this issue

                Dates

                • Created:
                  Updated: