Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      1) A TrafficController class that provides an implementation for traffic shaping using tc.
      2) A ResourceHandler implementation for OutboundBandwidth as a resource - isolation/enforcement using cgroups and tc.
      Show
      1) A TrafficController class that provides an implementation for traffic shaping using tc. 2) A ResourceHandler implementation for OutboundBandwidth as a resource - isolation/enforcement using cgroups and tc.

      Description

      In order to be able to isolate based on/enforce outbound traffic bandwidth limits, we need a mechanism to classify/shape network traffic in the nodemanager. For more information on the design, please see the attached design document in the parent JIRA.

      1. YARN-3366.007.patch
        82 kB
        Sidharta Seethana
      2. YARN-3366.006.patch
        79 kB
        Sidharta Seethana
      3. YARN-3366.005.patch
        79 kB
        Sidharta Seethana
      4. YARN-3366.004.patch
        79 kB
        Sidharta Seethana
      5. YARN-3366.003.patch
        81 kB
        Sidharta Seethana
      6. YARN-3366.002.patch
        82 kB
        Sidharta Seethana
      7. YARN-3366.001.patch
        78 kB
        Sidharta Seethana

        Activity

        Hide
        sidharta-s Sidharta Seethana added a comment -
        • classful, hierarchical traffic shaping with tc (and cgroups) works only for outbound network traffic - at this point this is no good way to do this on a per-container basis for incoming traffic. When packets have arrived and are ‘known’ (somehow) to be destined for a specific container, the network cost has already been incurred . We could penalize the (remote) application that is sending packets, at best .
        • (Outbound) Network bandwidth utilization is collected on a per-container basis ( in terms of ‘total bytes sent’ - which will have to be periodically queried ) but it has not been hooked up to any metrics collection mechanism. See the ‘getBytesSentPerContainer()’ function here : https://github.com/apache/hadoop/blob/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java#L213 . Again, this works only for outbound bandwidth and is restricted to containers only ( shuffle, HDFS, YARN NM traffic etc isn’t account for by this hook ) . Technically speaking, stat parsing based on tc output does gather outbound bandwidth utilization for all of the traffic shaping classes associated with a network interface and not just the containers, but through the resource handler we only expose utilization data for containers.

        While we haven't done specific microbenchmarks, I did remember running into some data in presentation by a redhat engineer - let me see if I can find it again.

        Show
        sidharta-s Sidharta Seethana added a comment - classful, hierarchical traffic shaping with tc (and cgroups) works only for outbound network traffic - at this point this is no good way to do this on a per-container basis for incoming traffic. When packets have arrived and are ‘known’ (somehow) to be destined for a specific container, the network cost has already been incurred . We could penalize the (remote) application that is sending packets, at best . (Outbound) Network bandwidth utilization is collected on a per-container basis ( in terms of ‘total bytes sent’ - which will have to be periodically queried ) but it has not been hooked up to any metrics collection mechanism. See the ‘getBytesSentPerContainer()’ function here : https://github.com/apache/hadoop/blob/branch-2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java#L213 . Again, this works only for outbound bandwidth and is restricted to containers only ( shuffle, HDFS, YARN NM traffic etc isn’t account for by this hook ) . Technically speaking, stat parsing based on tc output does gather outbound bandwidth utilization for all of the traffic shaping classes associated with a network interface and not just the containers, but through the resource handler we only expose utilization data for containers. While we haven't done specific microbenchmarks, I did remember running into some data in presentation by a redhat engineer - let me see if I can find it again.
        Hide
        srikanthkandula Srikanth Kandula added a comment -

        1) Does this also capture the network usage due to non containers? For eg. that due to evacuation or replication or data downloads?

        2) What about receive bandwidth?

        3) Perhaps i missed this above, but what are the overhead microbenchmark numbers re: added latency for normal packets and extra cpu usage overall due to sending packets through tc/ due to polling tc counters periodically?

        Show
        srikanthkandula Srikanth Kandula added a comment - 1) Does this also capture the network usage due to non containers? For eg. that due to evacuation or replication or data downloads? 2) What about receive bandwidth? 3) Perhaps i missed this above, but what are the overhead microbenchmark numbers re: added latency for normal packets and extra cpu usage overall due to sending packets through tc/ due to polling tc counters periodically?
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Jun Gong , The goal here is to allocate a maximum amount of bandwidth for YARN containers - irrespective of whether strict mode is enabled or not. Strict mode only determines per-container behavior and I am not sure it makes sense to extend its semantics to include other behavior. It is possible to set yarnBandwidthMbit to the same value as rootBandwidthMbit either by explicitly setting its value to be equal or not setting it at all ( in which case the behavior is to assign yarnBandwidthMbit to the value of rootBandwidthMbit).

        I recommend filing a separate JIRA in case you think we should be changing this, since the patch for this has been committed already.

        thanks.

        Show
        sidharta-s Sidharta Seethana added a comment - Jun Gong , The goal here is to allocate a maximum amount of bandwidth for YARN containers - irrespective of whether strict mode is enabled or not. Strict mode only determines per-container behavior and I am not sure it makes sense to extend its semantics to include other behavior. It is possible to set yarnBandwidthMbit to the same value as rootBandwidthMbit either by explicitly setting its value to be equal or not setting it at all ( in which case the behavior is to assign yarnBandwidthMbit to the value of rootBandwidthMbit). I recommend filing a separate JIRA in case you think we should be changing this, since the patch for this has been committed already. thanks.
        Hide
        hex108 Jun Gong added a comment -

        Sidharta Seethana Thank you for the explanation. Could we set YARNRootClass's ceil rate to yarnBandwidthMbit when 'strictMode'(which is configured through YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE) is set to true, otherwise set it to rootBandwidthMbit? If cluster admins require to set a maximum bandwidth for YARN, he/she could set strictMode to true.

        Show
        hex108 Jun Gong added a comment - Sidharta Seethana Thank you for the explanation. Could we set YARNRootClass's ceil rate to yarnBandwidthMbit when 'strictMode'(which is configured through YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE) is set to true, otherwise set it to rootBandwidthMbit? If cluster admins require to set a maximum bandwidth for YARN, he/she could set strictMode to true.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        hi Jun Gong ,

        The goal here is to ensure that we have a maximum assigned bandwidth for all YARN containers when cluster admins require this. The behavior you want (where YARN containers are allowed to use the total bandwidth) is possible by simply not configuring max YARN outbound bandwidth - in which case it defaults to the total bandwidth.

        thanks

        Show
        sidharta-s Sidharta Seethana added a comment - hi Jun Gong , The goal here is to ensure that we have a maximum assigned bandwidth for all YARN containers when cluster admins require this. The behavior you want (where YARN containers are allowed to use the total bandwidth) is possible by simply not configuring max YARN outbound bandwidth - in which case it defaults to the total bandwidth. thanks
        Hide
        hex108 Jun Gong added a comment -

        When addYARNRootClass in TrafficController#initializeState, could we set its ceil rate to rootBandwidthMbit instead of yarnBandwidthMbit? Then YARN could use more network bandwidth, and it will not affect defaultClass because its rate has been set to defaultClassBandwidthMbit.

        Show
        hex108 Jun Gong added a comment - When addYARNRootClass in TrafficController#initializeState, could we set its ceil rate to rootBandwidthMbit instead of yarnBandwidthMbit? Then YARN could use more network bandwidth, and it will not affect defaultClass because its rate has been set to defaultClassBandwidthMbit.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #2122 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2122/)
        YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2122 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2122/ ) YARN-3366 . Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/173/)
        YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #173 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/173/ ) YARN-3366 . Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #906 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/906/)
        YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #906 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/906/ ) YARN-3366 . Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #172 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/172/)
        YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #172 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/172/ ) YARN-3366 . Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #163 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/163/)
        YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #163 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/163/ ) YARN-3366 . Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2104 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2104/)
        YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2104 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2104/ ) YARN-3366 . Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java hadoop-yarn-project/CHANGES.txt
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #7642 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7642/)
        YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java
        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #7642 (See https://builds.apache.org/job/Hadoop-trunk-Commit/7642/ ) YARN-3366 . Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. (vinodkv: rev a100be685cc4521e9949589948219231aa5d2733) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficControlBandwidthHandlerImpl.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestTrafficController.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerModule.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficController.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/OutboundBandwidthResourceHandler.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TrafficControlBandwidthHandlerImpl.java
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Committed this to trunk and branch-2. Thanks Sid!

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Committed this to trunk and branch-2. Thanks Sid!
        Hide
        sidharta-s Sidharta Seethana added a comment -
        Show
        sidharta-s Sidharta Seethana added a comment - Here is the ticket : https://issues.apache.org/jira/browse/HADOOP-11869
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        +1 for the latest patch. Checking this in.

        Can you file a ticket for the checkstyle rules' issues?

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - +1 for the latest patch. Checking this in. Can you file a ticket for the checkstyle rules' issues?
        Hide
        sidharta-s Sidharta Seethana added a comment -

        The test failure is unrelated is unrelated to this patch. The checkstyle script and the rules in place need to be revisited - for example, I see warnings for "line too long" for import statements.

        Show
        sidharta-s Sidharta Seethana added a comment - The test failure is unrelated is unrelated to this patch. The checkstyle script and the rules in place need to be revisited - for example, I see warnings for "line too long" for import statements.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 15m 12s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 tests included 0m 0s The patch appears to include 3 new or modified test files.
        +1 whitespace 0m 0s The patch has no lines that end in whitespace.
        +1 javac 7m 37s There were no new javac warning messages.
        +1 javadoc 9m 59s There were no new javadoc warning messages.
        +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
        -1 checkstyle 5m 30s The applied patch generated 6 additional checkstyle issues.
        +1 install 1m 35s mvn install still works.
        +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
        +1 findbugs 2m 27s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
        +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api.
        -1 yarn tests 5m 48s Tests failed in hadoop-yarn-server-nodemanager.
            49m 29s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12727369/YARN-3366.007.patch
        Optional Tests javadoc javac unit findbugs checkstyle
        git revision trunk / 0ebe84d
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7463/artifact/patchprocess/checkstyle-result-diff.txt
        hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7463/artifact/patchprocess/testrun_hadoop-yarn-api.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7463/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7463/testReport/
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/7463//console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 15m 12s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 3 new or modified test files. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 javac 7m 37s There were no new javac warning messages. +1 javadoc 9m 59s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 5m 30s The applied patch generated 6 additional checkstyle issues. +1 install 1m 35s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 27s The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 yarn tests 0m 23s Tests passed in hadoop-yarn-api. -1 yarn tests 5m 48s Tests failed in hadoop-yarn-server-nodemanager.     49m 29s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12727369/YARN-3366.007.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0ebe84d checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/7463/artifact/patchprocess/checkstyle-result-diff.txt hadoop-yarn-api test log https://builds.apache.org/job/PreCommit-YARN-Build/7463/artifact/patchprocess/testrun_hadoop-yarn-api.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/7463/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/7463/testReport/ Console output https://builds.apache.org/job/PreCommit-YARN-Build/7463//console This message was automatically generated.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Attaching a new patch based on code-review feedback from Vinod Kumar Vavilapalli

        Show
        sidharta-s Sidharta Seethana added a comment - Attaching a new patch based on code-review feedback from Vinod Kumar Vavilapalli
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Hi Vinod Kumar Vavilapalli,

        conf.get("hadoop.tmp.dir"): We should write to the nmPrivate directories instead of /tmp.

        Digging in this further, it turns out that the change is far from trivial because of the way initialization works in the node manager today. I filed a separate JIRA to track this : https://issues.apache.org/jira/browse/YARN-3531 .

        I'll update the patch based on the rest of the feedback as discussed above.

        thanks

        Show
        sidharta-s Sidharta Seethana added a comment - Hi Vinod Kumar Vavilapalli , conf.get("hadoop.tmp.dir"): We should write to the nmPrivate directories instead of /tmp. Digging in this further, it turns out that the change is far from trivial because of the way initialization works in the node manager today. I filed a separate JIRA to track this : https://issues.apache.org/jira/browse/YARN-3531 . I'll update the patch based on the rest of the feedback as discussed above. thanks
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Thanks for your feedback, Vinod Kumar Vavilapalli . Responses inline :

        Add a bit of javadoc to TrafficControlBandwidthHandlerImpl ?

        Sure, I'll do that.

        You may want to mark YarnConfiguration.DEFAULT_NM_NETWORK_* constants also as Private, given the config knobs are.

        Will do.

        outbound-bandwidth-mbit -> node-outbound-bandwidth-mbit and outbound-bandwidth-yarn-mbit -> yarn-outbound-bandwidth-mbit?

        "outbound-bandwidth" is used a prefix for these config params. I would rather leave them as is for now - especially since they are likely to be changed in the future.

        conf.get("hadoop.tmp.dir"): We should write to the nmPrivate directories instead of /tmp.

        I'll figure out a way to do this.

        TrafficController
        Can the patterns used in checkIfAlreadyBootstrapped()/reacquireContainerClasses() etc be compiled once and reused?

        There isn't much value to doing this - these are executed once during NM startup and the regex patterns are not re-used.

        What is MIN_CONTAINER_CLASS_ID? Add some javadoc? May be for other constants?

        Some classids are already used to setup a hierarchy of classes. MIN_CONTAINER_CLASS_ID is used to ensure that there are no collisions.

        Fork off BatchBuilder into its own class?

        I am afraid this isn't possible/easy - The builder class is designed to make tc command generation easier and it very closely tied to the TrafficController class.

        TrafficControlBandwidthHandlerImpl: getBytesSentPerContainer() for future use?

        Yes, this is meant to provide metrics at a container level - when it is hooked in. I'll add a note.

        Show
        sidharta-s Sidharta Seethana added a comment - Thanks for your feedback, Vinod Kumar Vavilapalli . Responses inline : Add a bit of javadoc to TrafficControlBandwidthHandlerImpl ? Sure, I'll do that. You may want to mark YarnConfiguration.DEFAULT_NM_NETWORK_* constants also as Private, given the config knobs are. Will do. outbound-bandwidth-mbit -> node-outbound-bandwidth-mbit and outbound-bandwidth-yarn-mbit -> yarn-outbound-bandwidth-mbit? "outbound-bandwidth" is used a prefix for these config params. I would rather leave them as is for now - especially since they are likely to be changed in the future. conf.get("hadoop.tmp.dir"): We should write to the nmPrivate directories instead of /tmp. I'll figure out a way to do this. TrafficController Can the patterns used in checkIfAlreadyBootstrapped()/reacquireContainerClasses() etc be compiled once and reused? There isn't much value to doing this - these are executed once during NM startup and the regex patterns are not re-used. What is MIN_CONTAINER_CLASS_ID? Add some javadoc? May be for other constants? Some classids are already used to setup a hierarchy of classes. MIN_CONTAINER_CLASS_ID is used to ensure that there are no collisions. Fork off BatchBuilder into its own class? I am afraid this isn't possible/easy - The builder class is designed to make tc command generation easier and it very closely tied to the TrafficController class. TrafficControlBandwidthHandlerImpl: getBytesSentPerContainer() for future use? Yes, this is meant to provide metrics at a container level - when it is hooked in. I'll add a note.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Comments on the patch - The formatting in all the new files is a bit off.

        • Add a bit of javadoc to TrafficControlBandwidthHandlerImpl ?
        • You may want to mark YarnConfiguration.DEFAULT_NM_NETWORK_* constants also as Private, given the config knobs are.
        • outbound-bandwidth-mbit -> node-outbound-bandwidth-mbit and outbound-bandwidth-yarn-mbit -> yarn-outbound-bandwidth-mbit?
        • conf.get("hadoop.tmp.dir"): We should write to the nmPrivate directories instead of /tmp.
        • TrafficController
          • Can the patterns used in checkIfAlreadyBootstrapped()/reacquireContainerClasses() etc be compiled once and reused?
          • What is MIN_CONTAINER_CLASS_ID? Add some javadoc? May be for other constants?
        • Fork off BatchBuilder into its own class?
        • TrafficControlBandwidthHandlerImpl: getBytesSentPerContainer() for future use?
        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Comments on the patch - The formatting in all the new files is a bit off. Add a bit of javadoc to TrafficControlBandwidthHandlerImpl ? You may want to mark YarnConfiguration.DEFAULT_NM_NETWORK_* constants also as Private, given the config knobs are. outbound-bandwidth-mbit -> node-outbound-bandwidth-mbit and outbound-bandwidth-yarn-mbit -> yarn-outbound-bandwidth-mbit? conf.get("hadoop.tmp.dir") : We should write to the nmPrivate directories instead of /tmp. TrafficController Can the patterns used in checkIfAlreadyBootstrapped()/reacquireContainerClasses() etc be compiled once and reused? What is MIN_CONTAINER_CLASS_ID? Add some javadoc? May be for other constants? Fork off BatchBuilder into its own class? TrafficControlBandwidthHandlerImpl: getBytesSentPerContainer() for future use?
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Thanks, Junping Du !

        Show
        sidharta-s Sidharta Seethana added a comment - Thanks, Junping Du !
        Hide
        djp Junping Du added a comment -

        Thanks Sidharta Seethana for updating the patch!
        The latest patch LGTM. If no further comments from others, I will go ahead and commit it later today.

        Show
        djp Junping Du added a comment - Thanks Sidharta Seethana for updating the patch! The latest patch LGTM. If no further comments from others, I will go ahead and commit it later today.
        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12725923/YARN-3366.006.patch
        against trunk revision 80a2a12.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7364//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7364//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725923/YARN-3366.006.patch against trunk revision 80a2a12. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7364//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7364//console This message is automatically generated.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        All the -1s are not valid for this patch. It appears that build was triggered incorrectly. This patch has RM changes at all.

        Show
        sidharta-s Sidharta Seethana added a comment - All the -1s are not valid for this patch. It appears that build was triggered incorrectly. This patch has RM changes at all.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12725923/YARN-3366.006.patch
        against trunk revision 1fa8075.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

        org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
        org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
        org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections
        org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
        org.apache.hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore
        org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf
        org.apache.hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
        org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
        org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService
        org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
        org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
        org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore

        The following test timeouts occurred in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

        org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRTestTests
        org.apache.hadoop.yarn.server.resourcemanager.TestRMRestarTests
        org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
        org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7359//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7359//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7359//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725923/YARN-3366.006.patch against trunk revision 1fa8075. +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 2 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore org.apache.hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf org.apache.hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart org.apache.hadoop.yarn.server.resourcemanager.TestRMAdminService org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore The following test timeouts occurred in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRTestTests org.apache.hadoop.yarn.server.resourcemanager.TestRMRestarTests org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7359//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7359//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7359//console This message is automatically generated.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        hi Junping Du ,

        Uploading a new patch based on your feedback.

        Thanks,
        -Sidharta

        Show
        sidharta-s Sidharta Seethana added a comment - hi Junping Du , Uploading a new patch based on your feedback. Thanks, -Sidharta
        Hide
        sidharta-s Sidharta Seethana added a comment -

        hi Junping Du ,

        Thanks for review. Responses inline :
        1. About YarnConfiguration.java changes - the new configs are designated @Private - this is an alpha/preview feature and things could change in subsequent releases. Unfortunately, there doesn't seem to be a better way handle "preview" configurations. Please correct me if I am wrong.
        2. About making the interface config an array : At this point we only support traffic shaping on one interface (the primary interface being used for intra-cluster traffic). We could consider adding support for multiple interfaces in the future, but it is not supported for the time being.
        3. About empty resource handler chain : If there are no resource handlers configured, all the 'hooks' - bootstrap, preStart etc are no-ops. No exceptions are thrown.
        4. About TC_MODIFY_STATE - yes, I'll add the break. Good catch, thanks.
        5. About null check missing - I'll add this. I think this got deleted by accident - not sure how.
        6. About returning a boolean - sure, I'll remove the unused boolean return
        7. About MAX_CONTAINER_COUNT : Unfortunately, we cannot make this dynamic for the time being. We'll need to compute the bandwidth assigned during bootstrap/startup time. Once we have scheduling support for this resource type, this computation will go away. I will however, reduce the max count to 50. Please note that, unless strict usage is enabled, these is a soft-limit. A container is allowed to use more bandwidth if available.
        8. postComplete returning an op instead of null : this is by design. We currently do not have a way of batching operations in the container-executor binary apart from launch container - i.e a separate invocation is necessary anyway. Another reason for executing this op inline is that this is not in the performance critical "launch container" path.
        9. Debug log lines : Sure, I'll merge the lines. I think we should create a document somewhere with such unwritten rules ( or add this to the contributor document ) that this is a required convention.

        Show
        sidharta-s Sidharta Seethana added a comment - hi Junping Du , Thanks for review. Responses inline : 1. About YarnConfiguration.java changes - the new configs are designated @Private - this is an alpha/preview feature and things could change in subsequent releases. Unfortunately, there doesn't seem to be a better way handle "preview" configurations. Please correct me if I am wrong. 2. About making the interface config an array : At this point we only support traffic shaping on one interface (the primary interface being used for intra-cluster traffic). We could consider adding support for multiple interfaces in the future, but it is not supported for the time being. 3. About empty resource handler chain : If there are no resource handlers configured, all the 'hooks' - bootstrap, preStart etc are no-ops. No exceptions are thrown. 4. About TC_MODIFY_STATE - yes, I'll add the break. Good catch, thanks. 5. About null check missing - I'll add this. I think this got deleted by accident - not sure how. 6. About returning a boolean - sure, I'll remove the unused boolean return 7. About MAX_CONTAINER_COUNT : Unfortunately, we cannot make this dynamic for the time being. We'll need to compute the bandwidth assigned during bootstrap/startup time. Once we have scheduling support for this resource type, this computation will go away. I will however, reduce the max count to 50. Please note that, unless strict usage is enabled, these is a soft-limit. A container is allowed to use more bandwidth if available. 8. postComplete returning an op instead of null : this is by design. We currently do not have a way of batching operations in the container-executor binary apart from launch container - i.e a separate invocation is necessary anyway. Another reason for executing this op inline is that this is not in the performance critical "launch container" path. 9. Debug log lines : Sure, I'll merge the lines. I think we should create a document somewhere with such unwritten rules ( or add this to the contributor document ) that this is a required convention.
        Hide
        djp Junping Du added a comment -

        Thanks Sidharta Seethana for delivering the patch! A few comments so far:
        In YarnConfiguration.java,

        +  /** This setting controls if resource handling for network bandwidth is enabled **/
        +  /* Work in progress: This configuration parameter may be changed/removed in the future */
        +  @Private
        +  public static final String NM_NETWORK_RESOURCE_ENABLED =
        +      NM_NETWORK_RESOURCE_PREFIX + "enabled";
        +  /** Network as a resource is disabled by default **/
        +  public static final boolean DEFAULT_NM_NETWORK_RESOURCE_ENABLED = false;
        

        Why we are explicitly saying "Work in progress: This configuration parameter may be changed/removed in the future" here and in other places? These configuration properties, once go in and released, can only deprecated but cannot removed without major release update.

        +  public static final String NM_NETWORK_RESOURCE_INTERFACE =
        +      NM_NETWORK_RESOURCE_PREFIX + "interface";
        +  public static final String DEFAULT_NM_NETWORK_RESOURCE_INTERFACE = "eth0";
        

        Shall we support the case for multiple network interfaces? I know the user can do something like nic teaming in OS configuration layer. However, YARN shouldn't assume user have to do this. Isn't it? If so, better to update to String array.

        In LinuxContainerExecutor.java,

        +    try {
        +      resourceHandlerChain = ResourceHandlerModule
        +          .getConfiguredResourceHandlerChain(super.getConf());
        +      if (resourceHandlerChain != null) {
        +        resourceHandlerChain.bootstrap(super.getConf());
        +      }
        +    } catch (ResourceHandlerException e) {
        +      LOG.error("Failed to bootstrap configured resource subsystems! ", e);
        +      throw new IOException("Failed to bootstrap configured resource subsystems!");
        +    }
        

        If "NM_NETWORK_RESOURCE_ENABLED" = false, the resourceHandlerChain will still be initiated with empty handler but not null. So resourceHandlerChain.bootstrap() still get called which is not necessary and possible get exception thrown out. I think we should make sure we don't involve any operations if all resources configuration are disabled (NETWORK_RESOURCE only so far).
        In this case, may be we should make resourceHandlerChain to be null when NM_NETWORK_RESOURCE_ENABLED is false (assume other resources haven't onboard so far)? Also, other operations like: postComplete, reacquireContainer, etc. has the same issue.

        +          for (PrivilegedOperation op : ops) {
        +            switch (op.getOperationType()) {
        +              case ADD_PID_TO_CGROUP:
        +                resourceOps.add(op);
        +                break;
        +              case TC_MODIFY_STATE:
        +                tcCommandFile = op.getArguments().get(0);
        +              default:
        +                LOG.warn("PrivilegedOperation type unsupported in launch: "
        +                    + op.getOperationType());
        +                continue;
        +            }
        +          }
        

        We miss a break in case TC_MODIFY_STATE: ?

        In ResourceHandlerModule.java,

        +    if (cGroupsHandler == null) {
        +      synchronized (CGroupsHandler.class) {
        +        cGroupsHandler = new CGroupsHandlerImpl(conf,
        +            PrivilegedOperationExecutor.getInstance(conf));
        +      }
        +    }
        

        We miss a null check again inside of the synchronized (CGroupsHandler.class) block.

        +  private static boolean addHandlerIfNotNull(List<ResourceHandler> handlerList,
        +      ResourceHandler handler) {
        +    return (handler != null) && handlerList.add(handler);
        +  }
        

        Return a boolean value is not necessary?

        In TrafficControlBandwidthHandlerImpl.java,

        +  //In the absence of 'scheduling' support, we'll 'infer' the guaranteed
        +  //outbound bandwidth for each container based on this number. This will
        +  //likely go away once we add support on the RM for this resource type.
        +  private static final int MAX_CONTAINER_COUNT = 100;
        ...
        +    containerBandwidthMbit = (int) Math.ceil((double) yarnBandwidthMbit /
        +        MAX_CONTAINER_COUNT);
        

        Can we make containerBandwidthMbit being calculated dynamically by number of containers running on NM? If not, setting MAX_CONTAINER_COUNT to 100 sounds too large to me which could make containerBandwidthMbit smaller than necessary. Typically, containers on a powerful machine in production environment should be 10 - 50, may be we could set something like: 50 here?

        For postComplete(ContainerId containerId), we should return op instead of null?

        In TrafficController.java,

        +      if (LOG.isDebugEnabled()) {
        +        LOG.debug("TC state: ");
        +        LOG.debug(output);
        +      }
        ...
        +      if (LOG.isDebugEnabled()) {
        +        LOG.debug("classId -> bytes sent");
        +        LOG.debug(classIdBytesStats);
        +      }
        

        Can we merge two LOG.debug() into one line with %n if we want a new line in log? There are many other places need to be fixed too.

        Show
        djp Junping Du added a comment - Thanks Sidharta Seethana for delivering the patch! A few comments so far: In YarnConfiguration.java, + /** This setting controls if resource handling for network bandwidth is enabled **/ + /* Work in progress: This configuration parameter may be changed/removed in the future */ + @Private + public static final String NM_NETWORK_RESOURCE_ENABLED = + NM_NETWORK_RESOURCE_PREFIX + "enabled" ; + /** Network as a resource is disabled by default **/ + public static final boolean DEFAULT_NM_NETWORK_RESOURCE_ENABLED = false ; Why we are explicitly saying "Work in progress: This configuration parameter may be changed/removed in the future" here and in other places? These configuration properties, once go in and released, can only deprecated but cannot removed without major release update. + public static final String NM_NETWORK_RESOURCE_INTERFACE = + NM_NETWORK_RESOURCE_PREFIX + " interface " ; + public static final String DEFAULT_NM_NETWORK_RESOURCE_INTERFACE = "eth0" ; Shall we support the case for multiple network interfaces? I know the user can do something like nic teaming in OS configuration layer. However, YARN shouldn't assume user have to do this. Isn't it? If so, better to update to String array. In LinuxContainerExecutor.java, + try { + resourceHandlerChain = ResourceHandlerModule + .getConfiguredResourceHandlerChain( super .getConf()); + if (resourceHandlerChain != null ) { + resourceHandlerChain.bootstrap( super .getConf()); + } + } catch (ResourceHandlerException e) { + LOG.error( "Failed to bootstrap configured resource subsystems! " , e); + throw new IOException( "Failed to bootstrap configured resource subsystems!" ); + } If "NM_NETWORK_RESOURCE_ENABLED" = false, the resourceHandlerChain will still be initiated with empty handler but not null. So resourceHandlerChain.bootstrap() still get called which is not necessary and possible get exception thrown out. I think we should make sure we don't involve any operations if all resources configuration are disabled (NETWORK_RESOURCE only so far). In this case, may be we should make resourceHandlerChain to be null when NM_NETWORK_RESOURCE_ENABLED is false (assume other resources haven't onboard so far)? Also, other operations like: postComplete, reacquireContainer, etc. has the same issue. + for (PrivilegedOperation op : ops) { + switch (op.getOperationType()) { + case ADD_PID_TO_CGROUP: + resourceOps.add(op); + break ; + case TC_MODIFY_STATE: + tcCommandFile = op.getArguments().get(0); + default : + LOG.warn( "PrivilegedOperation type unsupported in launch: " + + op.getOperationType()); + continue ; + } + } We miss a break in case TC_MODIFY_STATE: ? In ResourceHandlerModule.java, + if (cGroupsHandler == null ) { + synchronized (CGroupsHandler.class) { + cGroupsHandler = new CGroupsHandlerImpl(conf, + PrivilegedOperationExecutor.getInstance(conf)); + } + } We miss a null check again inside of the synchronized (CGroupsHandler.class) block. + private static boolean addHandlerIfNotNull(List<ResourceHandler> handlerList, + ResourceHandler handler) { + return (handler != null ) && handlerList.add(handler); + } Return a boolean value is not necessary? In TrafficControlBandwidthHandlerImpl.java, + //In the absence of 'scheduling' support, we'll 'infer' the guaranteed + //outbound bandwidth for each container based on this number. This will + //likely go away once we add support on the RM for this resource type. + private static final int MAX_CONTAINER_COUNT = 100; ... + containerBandwidthMbit = ( int ) Math .ceil(( double ) yarnBandwidthMbit / + MAX_CONTAINER_COUNT); Can we make containerBandwidthMbit being calculated dynamically by number of containers running on NM? If not, setting MAX_CONTAINER_COUNT to 100 sounds too large to me which could make containerBandwidthMbit smaller than necessary. Typically, containers on a powerful machine in production environment should be 10 - 50, may be we could set something like: 50 here? For postComplete(ContainerId containerId), we should return op instead of null? In TrafficController.java, + if (LOG.isDebugEnabled()) { + LOG.debug( "TC state: " ); + LOG.debug(output); + } ... + if (LOG.isDebugEnabled()) { + LOG.debug( "classId -> bytes sent" ); + LOG.debug(classIdBytesStats); + } Can we merge two LOG.debug() into one line with %n if we want a new line in log? There are many other places need to be fixed too.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        The findbugs warnings are fixed. The test failure is unrelated to this patch.

        Show
        sidharta-s Sidharta Seethana added a comment - The findbugs warnings are fixed. The test failure is unrelated to this patch.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12725149/YARN-3366.005.patch
        against trunk revision b9b832a.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7325//testReport/
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7325//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725149/YARN-3366.005.patch against trunk revision b9b832a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7325//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7325//console This message is automatically generated.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Uploaded a new patch with findbugs warnings fixed.

        Show
        sidharta-s Sidharta Seethana added a comment - Uploaded a new patch with findbugs warnings fixed.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12725133/YARN-3366.004.patch
        against trunk revision 838b06a.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        -1 findbugs. The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

        Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7323//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7323//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
        Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7323//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725133/YARN-3366.004.patch against trunk revision 838b06a. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/7323//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/7323//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7323//console This message is automatically generated.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Uploading rebased patch

        Show
        sidharta-s Sidharta Seethana added a comment - Uploading rebased patch
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Thanks, Varun Vasudev . LinuxContainerExecutor and CgroupsLCEResourcesHandler will have to be refactored once these patches are in - I'll file a ticket for this.

        Show
        sidharta-s Sidharta Seethana added a comment - Thanks, Varun Vasudev . LinuxContainerExecutor and CgroupsLCEResourcesHandler will have to be refactored once these patches are in - I'll file a ticket for this.
        Hide
        vvasudev Varun Vasudev added a comment -

        +1 pending jenkins. I think you should about filing a ticket about refactoring launchContainer.

        Show
        vvasudev Varun Vasudev added a comment - +1 pending jenkins. I think you should about filing a ticket about refactoring launchContainer.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Uploading patch incorporating code review feedback.

        Show
        sidharta-s Sidharta Seethana added a comment - Uploading patch incorporating code review feedback.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Thanks for the review, Varun Vasudev . Responses inline :

        1. I'll fix this. This is an artifact of differences between trunk/branch-2
        (repeated) 1. I think these are useful log lines that specify change in behavior due to settings/system state etc. I'll clarify/improve the log messages.
        2. good catch, I'll fix it. Tests ran fine because WARN logging was enabled.
        3. I'll fix the comments' location. The exception used to exist before but was causing bootstrapping issues. I left it in there along with an explanation for why it shouldn't be thrown. I'll remove it and modify comments.
        4. Intellij warns me about this too - but I had left it in there for clarity/consistency with the earlier code block - I believe it makes the code a bit more readable. I would prefer to leave it in place.
        5. I'll fix this
        6. I'll fix this
        7. why? compiler optimization?
        8. I'll fix this.
        9. I'll fix this.
        10. I'll fix this.
        11. I'll fix this - though I don't believe the merging always helps for error/warn metrics
        12. I'll fix this.
        13. Not trivially, would refactoring launchContainer.

        Show
        sidharta-s Sidharta Seethana added a comment - Thanks for the review, Varun Vasudev . Responses inline : 1. I'll fix this. This is an artifact of differences between trunk/branch-2 (repeated) 1. I think these are useful log lines that specify change in behavior due to settings/system state etc. I'll clarify/improve the log messages. 2. good catch, I'll fix it. Tests ran fine because WARN logging was enabled. 3. I'll fix the comments' location. The exception used to exist before but was causing bootstrapping issues. I left it in there along with an explanation for why it shouldn't be thrown. I'll remove it and modify comments. 4. Intellij warns me about this too - but I had left it in there for clarity/consistency with the earlier code block - I believe it makes the code a bit more readable. I would prefer to leave it in place. 5. I'll fix this 6. I'll fix this 7. why? compiler optimization? 8. I'll fix this. 9. I'll fix this. 10. I'll fix this. 11. I'll fix this - though I don't believe the merging always helps for error/warn metrics 12. I'll fix this. 13. Not trivially, would refactoring launchContainer.
        Hide
        vvasudev Varun Vasudev added a comment -

        Thanks for the patch Sidharta Seethana! Feedback below.

        1. In YarnConfiguration.java
             /**
          -   * True if linux-container-executor should limit itself to one user
          +   * If linux-container-executor should limit itself to one user
              * when running in non-secure mode.
              */
          -  public static final String NM_NONSECURE_MODE_LIMIT_USERS = NM_PREFIX +
          +  public static final String NM_NONSECURE_MODE_LIMIT_USERS= NM_PREFIX +
                "linux-container-executor.nonsecure-mode.limit-users";
          
          -  public static final boolean DEFAULT_NM_NONSECURE_MODE_LIMIT_USERS = true;
          +  public static final boolean DEFAULT_NM_NONSECURE_MODE_LIMIT_USERS = true;
          

        It looks like these are unnecessary changes. Can you please remove them?

        1. In TrafficController.java
          if (LOG.isInfoEnabled()) {
            LOG.info("NM recovery is not enabled.");
          }
          
          if (LOG.isInfoEnabled()) {
            LOG.info("TC configuration is incomplete.");
          }
          

          Can you change these to debug? It doesn't seem to be something that needs to be logged by the class.

        2. In TrafficController.java
          else {
            if (LOG.isWarnEnabled()) {
              String logLine = new StringBuffer("Failed to match regex: ")
                        .append(regex).append(" Current state: ").append(state).toString();
              LOG.warn(logLine);
              return false;
            }
          }
          

          Shouldn't the return be outside the warn enabled check?

        3. In TrafficController.java
          //This could happen if the interface is already in its default state.
          //Ignoring.
          //throw new ResourceHandlerException("Failed to wipe tc state", e);
          

          The comments are in a different block than the warn message. Also, the commented throw is confusing.

        4. Minor nit - In TrafficController.java, function parseStatsString, the continue isn't really required
        5. In TrafficControlBandwidthHandlerImpl.java - Unused import import com.google.common.annotations.VisibleForTesting
        6. In TrafficControlBandwidthHandlerImpl.java
          LOG.info("strict mode is set to : " + strictMode);
          
          LOG.info("Attempting to reacquire classId for container: " +
            containerIdStr);
          

          Change levels to debug?

        7. In TrafficControlBandwidthHandlerImpl.java
          String opArg = new StringBuffer(PrivilegedOperation.CGROUP_ARG_PREFIX)
                  .append(tasksFile).toString();
          

          You can use the String class itself instead of StringBuffer?

        8. In TrafficControlBandwidthHandlerImpl.java
          if (LOG.isWarnEnabled()) {
            LOG.warn("teardown(): Nothing to do");
          }
          

          Why are you logging a warning?

        9. In TestTrafficControlBandwidthHandlerImpl.java and TestTrafficController.java
          Assert.assertTrue("Caught unexpected ResourceHandlerException!", false);
          

          User Assert.fail? This pattern is used in multiple places.

        10. In LinuxContainerExecutor.java.java
          } catch (ResourceHandlerException e) {
          +          if (LOG.isWarnEnabled()) {
          +            LOG.warn("ResourceHandlerChain.reacquireContainer failed for " +
          +                "containerId: " + containerId);
          +          }
          

          Can you add the exception to the warn message?

        11. In LinuxContainerExecutor.java
          } catch (ResourceHandlerException e) {
                  if (LOG.isWarnEnabled()) {
                    LOG.warn(e);
                    LOG.warn("ResourceHandlerChain.postComplete failed for " +
                        "containerId: " + containerId);
                  }
          }
          

          Merge the warn messages.

        12. In LinuxContainerExecutor.java
          +    command.addAll(Arrays.asList(containerExecutorExe,
          

          Remove the extra space added.

        13. In LinuxContainerExecutor.java
          +    String tcCommandFile = null;
          +
          +    try {
          +      if (resourceHandlerChain != null) {
          +        List<PrivilegedOperation> ops = resourceHandlerChain
          +            .preStart(container);
          +
          +        if (ops != null) {
          +          List<PrivilegedOperation> resourceOps = new ArrayList<>();
          +
          +          resourceOps.add(new PrivilegedOperation
          +              (PrivilegedOperation.OperationType.ADD_PID_TO_CGROUP,
          +                  resourcesOptions));
          +
          +          for (PrivilegedOperation op : ops) {
          +            switch (op.getOperationType()) {
          +              case ADD_PID_TO_CGROUP:
          +                resourceOps.add(op);
          +                break;
          +              case TC_MODIFY_STATE:
          +                tcCommandFile = op.getArguments().get(0);
          +              default:
          +                if (LOG.isWarnEnabled()) {
          +                  LOG.warn("PrivilegedOperation type unsupported in launch: "
          +                      + op.getOperationType());
          +                }
          +                continue;
          +            }
          +          }
          +
          +          if (resourceOps.size() > 1) {
          +            //squash resource operations
          +            try {
          +              PrivilegedOperation operation = PrivilegedOperationExecutor
          +                  .squashCGroupOperations(resourceOps);
          +              resourcesOptions = operation.getArguments().get(0);
          +            } catch (PrivilegedOperationException e) {
          +              if (LOG.isErrorEnabled()) {
          +                LOG.error("Failed to squash cgroup operations!", e);
          +              }
          +
          +              throw new ResourceHandlerException("Failed to squash cgroup operations!");
          +            }
          +          }
          +        }
          +      }
          +    } catch (ResourceHandlerException e) {
          +      if (LOG.isErrorEnabled()) {
          +        LOG.error("ResourceHandlerChain.preStart() failed!", e);
          +      }
          +
          +      throw new IOException("ResourceHandlerChain.preStart() failed!");
          +    }
          

          Can this block be made a method?

        Show
        vvasudev Varun Vasudev added a comment - Thanks for the patch Sidharta Seethana ! Feedback below. In YarnConfiguration.java /** - * True if linux-container-executor should limit itself to one user + * If linux-container-executor should limit itself to one user * when running in non-secure mode. */ - public static final String NM_NONSECURE_MODE_LIMIT_USERS = NM_PREFIX + + public static final String NM_NONSECURE_MODE_LIMIT_USERS= NM_PREFIX + "linux-container-executor.nonsecure-mode.limit-users"; - public static final boolean DEFAULT_NM_NONSECURE_MODE_LIMIT_USERS = true; + public static final boolean DEFAULT_NM_NONSECURE_MODE_LIMIT_USERS = true; It looks like these are unnecessary changes. Can you please remove them? In TrafficController.java if (LOG.isInfoEnabled()) { LOG.info("NM recovery is not enabled."); } if (LOG.isInfoEnabled()) { LOG.info("TC configuration is incomplete."); } Can you change these to debug? It doesn't seem to be something that needs to be logged by the class. In TrafficController.java else { if (LOG.isWarnEnabled()) { String logLine = new StringBuffer("Failed to match regex: ") .append(regex).append(" Current state: ").append(state).toString(); LOG.warn(logLine); return false; } } Shouldn't the return be outside the warn enabled check? In TrafficController.java //This could happen if the interface is already in its default state. //Ignoring. //throw new ResourceHandlerException("Failed to wipe tc state", e); The comments are in a different block than the warn message. Also, the commented throw is confusing. Minor nit - In TrafficController.java, function parseStatsString, the continue isn't really required In TrafficControlBandwidthHandlerImpl.java - Unused import import com.google.common.annotations.VisibleForTesting In TrafficControlBandwidthHandlerImpl.java LOG.info("strict mode is set to : " + strictMode); LOG.info("Attempting to reacquire classId for container: " + containerIdStr); Change levels to debug? In TrafficControlBandwidthHandlerImpl.java String opArg = new StringBuffer(PrivilegedOperation.CGROUP_ARG_PREFIX) .append(tasksFile).toString(); You can use the String class itself instead of StringBuffer? In TrafficControlBandwidthHandlerImpl.java if (LOG.isWarnEnabled()) { LOG.warn("teardown(): Nothing to do"); } Why are you logging a warning? In TestTrafficControlBandwidthHandlerImpl.java and TestTrafficController.java Assert.assertTrue("Caught unexpected ResourceHandlerException!", false); User Assert.fail? This pattern is used in multiple places. In LinuxContainerExecutor.java.java } catch (ResourceHandlerException e) { + if (LOG.isWarnEnabled()) { + LOG.warn("ResourceHandlerChain.reacquireContainer failed for " + + "containerId: " + containerId); + } Can you add the exception to the warn message? In LinuxContainerExecutor.java } catch (ResourceHandlerException e) { if (LOG.isWarnEnabled()) { LOG.warn(e); LOG.warn("ResourceHandlerChain.postComplete failed for " + "containerId: " + containerId); } } Merge the warn messages. In LinuxContainerExecutor.java + command.addAll(Arrays.asList(containerExecutorExe, Remove the extra space added. In LinuxContainerExecutor.java + String tcCommandFile = null; + + try { + if (resourceHandlerChain != null) { + List<PrivilegedOperation> ops = resourceHandlerChain + .preStart(container); + + if (ops != null) { + List<PrivilegedOperation> resourceOps = new ArrayList<>(); + + resourceOps.add(new PrivilegedOperation + (PrivilegedOperation.OperationType.ADD_PID_TO_CGROUP, + resourcesOptions)); + + for (PrivilegedOperation op : ops) { + switch (op.getOperationType()) { + case ADD_PID_TO_CGROUP: + resourceOps.add(op); + break; + case TC_MODIFY_STATE: + tcCommandFile = op.getArguments().get(0); + default: + if (LOG.isWarnEnabled()) { + LOG.warn("PrivilegedOperation type unsupported in launch: " + + op.getOperationType()); + } + continue; + } + } + + if (resourceOps.size() > 1) { + //squash resource operations + try { + PrivilegedOperation operation = PrivilegedOperationExecutor + .squashCGroupOperations(resourceOps); + resourcesOptions = operation.getArguments().get(0); + } catch (PrivilegedOperationException e) { + if (LOG.isErrorEnabled()) { + LOG.error("Failed to squash cgroup operations!", e); + } + + throw new ResourceHandlerException("Failed to squash cgroup operations!"); + } + } + } + } + } catch (ResourceHandlerException e) { + if (LOG.isErrorEnabled()) { + LOG.error("ResourceHandlerChain.preStart() failed!", e); + } + + throw new IOException("ResourceHandlerChain.preStart() failed!"); + } Can this block be made a method?
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Uploading a patch that includes changes to YarnConfiguration.java

        Show
        sidharta-s Sidharta Seethana added a comment - Uploading a patch that includes changes to YarnConfiguration.java
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Since this patch requires uncommitted changes from https://issues.apache.org/jira/browse/YARN-3443, I am not submitting this patch to a pre-commit build for the time being.

        Show
        sidharta-s Sidharta Seethana added a comment - Since this patch requires uncommitted changes from https://issues.apache.org/jira/browse/YARN-3443 , I am not submitting this patch to a pre-commit build for the time being.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        Attaching a patch with an implementation of traffic classification/shaping for traffic originating from YARN containers. This patch depends on changes/patches from https://issues.apache.org/jira/browse/YARN-3365 and https://issues.apache.org/jira/browse/YARN-3443

        Show
        sidharta-s Sidharta Seethana added a comment - Attaching a patch with an implementation of traffic classification/shaping for traffic originating from YARN containers. This patch depends on changes/patches from https://issues.apache.org/jira/browse/YARN-3365 and https://issues.apache.org/jira/browse/YARN-3443

          People

          • Assignee:
            sidharta-s Sidharta Seethana
            Reporter:
            sidharta-s Sidharta Seethana
          • Votes:
            0 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development