[KUDU-2738] linked_list-test occasionally fails with webserver port bind failure: address already in use

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Cannot Reproduce
• Affects Version/s: 1.9.0
• Fix Version/s: NA
• Component/s: test
• Assignee: Unassigned
• Reporter: Mike Percy
• Created: 07/Mar/19 19:36
• Updated: 03/Jun/20 18:12
• Resolved: 03/Jun/20 18:12

Description

Occasionally I see linked_list-test fail with the following error on Linux in an automated test environment:

E0306 23:35:25.207222 19523 webserver.cc:369] Webserver: set_ports_option: cannot bind to 127.14.25.194:49008: 98 (Address already in use)
W0306 23:35:25.207244 19523 net_util.cc:457] Trying to use lsof to find any processes listening on 0.0.0.0:49008
I0306 23:35:25.207249 19523 net_util.cc:460] $ export PATH=$PATH:/usr/sbin ; lsof -n -i 'TCP:49008' -sTCP:LISTEN ; for pid in $(lsof -F p -n -i 'TCP:49008' -sTCP:LISTEN | grep p | cut -f 2 -dp) ; do while [ $pid -gt 1 ] ; do ps h -fp $pid ; stat=($(</proc/$pid/stat)) ; pid=${stat[3]} ; done ; done
...
W0306 23:35:25.583075 19523 net_util.cc:467]
F0306 23:35:25.583206 19523 tablet_server_main.cc:89] Check failed: _s.ok() Bad status: Runtime error: Webserver: could not start on address 127.14.25.194:49008: set_ports_option: cannot bind to 127.14.25.194:49008: 98 (Address already in use)

I am not sure what would have bound to 0.0.0.0:49008 for a short period of time, or used 126.14.25.194:49008 as an ephemeral address / port pair since it's such a unique loopback IP address.