[MESOS-6420] Mesos Agent leaking sockets when port mapping network isolator is ON

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.0.2
• Fix Version/s: 1.0.2, 1.1.0, 1.2.0
• Component/s: agent, containerization, network
• Target Version/s: 1.0.2, 1.1.0, 1.2.0
• Assignee: Unassigned
• Reporter: Santhosh Shanmugham
• Shepherd: Jie Yu
• Created: 19/Oct/16 21:00
• Updated: 26/Apr/17 16:53
• Resolved: 20/Oct/16 15:57

Description

Mesos Agent leaks one socket per task launched and eventually runs out of sockets. We were able to track it down to the network isolator (port_mapping.cpp). When we turned off the port mapping isolator no file descriptors where leaked. The leaked fd is a SOCK_STREAM socket.

Leaked Sockets:
$ sudo lsof -p $(pgrep -u root -o -f /usr/local/sbin/mesos-slave) -nP | grep "can't"
[sudo] password for sshanmugham:
mesos-sla 57688 root 19u sock 0,6 0t0 2993216948 can't identify protocol
mesos-sla 57688 root 27u sock 0,6 0t0 2993216468 can't identify protocol

Extract from strace:

...
[pid 57701] 19:14:02.493718 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.494395 close(19) = 0
[pid 57701] 19:14:02.494448 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.494844 close(19) = 0
[pid 57701] 19:14:02.494913 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.495565 close(19) = 0
[pid 57701] 19:14:02.495617 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.496072 close(19) = 0
[pid 57701] 19:14:02.496128 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.496758 close(19) = 0
[pid 57701] 19:14:02.496812 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.497270 close(19) = 0
[pid 57701] 19:14:02.497319 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.497698 close(19) = 0
[pid 57701] 19:14:02.497750 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.498407 close(19) = 0
[pid 57701] 19:14:02.498456 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.498899 close(19) = 0
[pid 57701] 19:14:02.498963 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 63682] 19:14:02.499091 close(18 <unfinished ...>
[pid 57701] 19:14:02.499634 close(19) = 0
[pid 57701] 19:14:02.499689 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.500044 close(19) = 0
[pid 57701] 19:14:02.500093 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.500734 close(19) = 0
[pid 57701] 19:14:02.500782 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.501271 close(19) = 0
[pid 57701] 19:14:02.501339 socket(PF_NETLINK, SOCK_RAW, 0) = 19
[pid 57701] 19:14:02.502030 close(19) = 0
[pid 57701] 19:14:02.502101 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 19
...

...
[pid 57691] 19:18:03.461022 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 6138] 19:18:03.461345 open("/etc/selinux/config", O_RDONLY <unfinished ...>
[pid 57691] 19:18:03.461460 close(27) = 0
[pid 57691] 19:18:03.461520 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 6138] 19:18:03.461632 close(3 <unfinished ...>
[pid 6138] 19:18:03.461781 open("/proc/mounts", O_RDONLY <unfinished ...>
[pid 6138] 19:18:03.462190 close(3 <unfinished ...>
[pid 57691] 19:18:03.462374 close(27) = 0
[pid 57691] 19:18:03.462430 socket(PF_NETLINK, SOCK_RAW, 0 <unfinished ...>
[pid 6138] 19:18:03.462456 open("/proc/net/psched", O_RDONLY <unfinished ...>
[pid 6138] 19:18:03.462678 close(3 <unfinished ...>
[pid 6138] 19:18:03.462915 open("/etc/libnl/classid", O_RDONLY <unfinished ...>
[pid 57691] 19:18:03.463046 close(27) = 0
[pid 57691] 19:18:03.463111 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 6138] 19:18:03.463225 close(3 <unfinished ...>
[pid 57691] 19:18:03.463845 close(27) = 0
[pid 57691] 19:18:03.463911 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.464604 close(27) = 0
[pid 57691] 19:18:03.464664 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.465074 close(27) = 0
[pid 57691] 19:18:03.465132 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.465862 close(27) = 0
[pid 57691] 19:18:03.465928 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.466713 close(27) = 0
[pid 57691] 19:18:03.466780 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.467472 close(27) = 0
[pid 57691] 19:18:03.467524 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.468012 close(27) = 0
[pid 57691] 19:18:03.468075 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.468799 close(27) = 0
[pid 57691] 19:18:03.468950 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.469505 close(27) = 0
[pid 57691] 19:18:03.469578 socket(PF_NETLINK, SOCK_RAW, 0) = 27
[pid 57691] 19:18:03.470301 close(27) = 0
[pid 57691] 19:18:03.470353 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 27
...

The last socket the was created never has a corresponding close().