Had a test failure last night in KerberosProxyIT. It seems the following happened
- Bind a server socket using '0' (pull a free port from the ephemeral range)
- Determine the actual port which was bound
- Close the socket
- Another process grabbed the port we had bound
- Attempt to start the proxy on that port
The proxy failed to start because the port we provided was already taken. This stems from a problem that we can't have the Proxy bind to any free port and figure out where it started (what port it actually bound); maybe we could provide configuration to advertise this in ZooKeeper (will file another issue). For this test, if we fail to connect to the proxy and the proxy is dead, we can try to re-start the proxy with a different port. Given how rare this scenario is, hopefully retrying it takes the chance it happens again to effectively 0%.