[MESOS-2332] Report per-container metrics for network bandwidth throttling

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.23.0
• Component/s: containerization
• Labels: features twitter
• Target Version/s: 0.23.0
• Epic Link: Container Network Isolation
• Sprint: Twitter Mesos Q1 Sprint 2, Twitter Mesos Q1 Sprint 3, Twitter Mesos Q1 Sprint 4, Twitter Mesos Q1 Sprint 5, Twitter Mesos Q1 Sprint 6, Twitter Q2 Sprint 1 - 4/13, Twitter Q2 Sprint 2
• Story Points: 5
• Assignee: Paul Brett
• Reporter: Paul Brett
• Shepherd: Jie Yu
• Created: 10/Feb/15 01:53
• Updated: 26/Apr/17 16:54
• Resolved: 18/Jun/15 00:18

Description

Export metrics from the network isolation to identify scope and duration of container throttling.

Packet loss can be identified from the overlimits and requeues fields of the htb qdisc report for the virtual interface, e.g.

$ tc -s -d qdisc show dev mesos19223
qdisc pfifo_fast 0: root refcnt 2 bands 3 priomap  1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
 Sent 158213287452 bytes 1030876393 pkt (dropped 0, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0
qdisc ingress ffff: parent ffff:fff1 ----------------
 Sent 119381747824 bytes 1144549901 pkt (dropped 2044879, overlimits 0 requeues 0)
 backlog 0b 0p requeues 0

Note that since a packet can be examined multiple times before transmission, overlimits can exceed total packets sent.

Add to the port_mapping isolator usage() and the container statistics protobuf. Carefully consider the naming (esp tx/rx) + commenting of the protobuf fields so it's clear what these represent and how they are different to the existing dropped packet counts from the network stack.

Issue Links

• depends upon: MESOS-2665 Fix queuing discipline wrapper in linux/routing/queueing
• depends upon: MESOS-2750 Extend queueing discipline wrappers to expose network isolator statistics
• depends upon: MESOS-2752 Add HTB queueing discipline wrapper class
• relates to: MESOS-1598 Add advanced shaping controls to routing library