[HBASE-24155] When running the tests, a tremendous number of connections are put into TIME_WAIT.

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Information Provided
• Component/s: test
• Assignee: Unassigned
• Reporter: Mark Robert Miller
• Created: 09/Apr/20 14:12
• Updated: 22/May/20 05:05
• Resolved: 13/May/20 13:24

Description

When you run the test suite and monitor the number of connections in TIME_WAIT, it appears that a very large number of connections do not end up with a proper connection close lifecycle or perhaps proper reuse.

Given connections can stay in TIME_WAIT from 1-4 minutes depending on OS/Env, running the tests faster or with more tests in parallel increases the TIME_WAIT connection buildup. Some tests spin up a very, very large number of connections and if the wrong ones run at the same time, this can also greatly increase the number of connections put into TIME_WAIT. This can have a dramatic affect on performance (as it can take longer to create a new connection) or flat out fail or timeout.

In my experience, a much, much smaller number of connections in a test suite would end up in TIME_WAIT when connection handling is all correct.

Notes to come in comments below.

Issue Links

• is related to: HBASE-9393 Region Server fails to properly close socket resulting in many CLOSE_WAIT to Data Nodes