[HADOOP-18105] Implement a variant of ElasticByteBufferPool which uses weak references for garbage collection.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.5
• Component/s: common, fs
• Labels: pull-request-available
• Assignee: Mukund Thakur
• Reporter: Mukund Thakur
• Created: 01/Feb/22 13:52
• Updated: 18/Nov/24 05:06
• Resolved: 16/Jun/22 20:12

Description

Currently in hadoop codebase, we have two classes which implements byte buffers pooling.

One is ElasticByteBufferPool which doesn't use weak references and thus could cause memory leaks in production environment. 

Other is DirectBufferPool which uses weak references but doesn't support caller's preference for either on-heap or off-heap buffers. 

 

The idea is to create an improved version of ElasticByteBufferPool by subclassing it ( as it is marked as public and stable and used widely in hdfs ) with essential functionalities required for effective buffer pooling. This is important for the parent Vectored IO work.

Issue Links

• relates to: HDFS-17670 HDFS Client's direct memory leaks with erasure coding enabled
• links to: GitHub Pull Request #4263