[ACCUMULO-4787] Numerous leaked CLOSE_WAIT threads from TabletServer

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.8.1
• Fix Version/s: 1.7.4, 1.9.0
• Labels: pull-request-available

Environment

• Ubuntu 14.04
• HDFS 2.6.0 and 2.5.0 (in the middle of an upgrade cycle)
• ZooKeeper 3.4.6
• Accumulo 1.8.1
• HotSpot 1.8.0_121

• Assignee: Adam J. Shook
• Reporter: Adam J. Shook
• Created: 26/Jan/18 17:05
• Updated: 23/Apr/19 16:13
• Resolved: 14/Feb/18 20:28

Description

I'm running into an issue across all environments where TabletServers are occupying a large number of ports in a CLOSED_WAIT state writing to a DataNode at port 50010.  I'm seeing numbers from around 12,000 to 20,000 ports.  In some instances, there were over 68k and it was affecting other applications from getting a free port and they would fail to start (which is how we found this in the first place).

Issue Links

• is related to: ACCUMULO-4816 DfsLogger.readHeaderAndReturnStream should return a Closeable
• links to: GitHub Pull Request #369