[HBASE-28666] Dropping unclosed WALTailingReaders leads to leaked sockets

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.7.0, 3.0.0-beta-2, 2.6.1
• Component/s: Replication, wal
• Labels: pull-request-available
• Hadoop Flags: Reviewed
• Assignee: Charles Connell
• Reporter: Charles Connell
• Created: 14/Jun/24 15:34
• Updated: 17/Jun/24 21:58
• Resolved: 16/Jun/24 02:36

Description

WALEntryStream#prepareReader() will, in some cases, reach the line

reader = WALFactory.createTailingReader(fs, nextPath, conf, currentPositionOfEntry > 0 ? currentPositionOfEntry : -1);

when reader is non-null. In this case, the old object pointed to by reader becomes un-referenced and is garbage-collected. However, that object was never closed.

At Hubspot we see the effects of this when doing tests that use inter-cluster replication. Machines in the source cluster experience a build-up of sockets. Eventually this causes the machine to run out of TCP kernel memory and start dropping packets. The only workaround currently is to restart the RegionServer process.

I have found that simply putting

closeReader();

immediately before the line quoted above appears to resolve the issue and causes no obvious problems. However, I'm still developing a proper test for this fix.

Issue Links

• is caused by: HBASE-27715 Refactoring the long tryAdvanceEntry method in WALEntryStream
• links to: GitHub Pull Request #5994