[HBASE-24051] When the data directory is set to EC,the amount of storefiles reaches a relatively high level,heap Occupancy Percent is above heap occupancy alarm watermark,resulting in frequent gc and eventual failure

• Type: Bug
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.0.1
• Hadoop Flags: Reviewed
• Assignee: shenshengli
• Reporter: shenshengli
• Created: 26/Mar/20 03:34
• Updated: 08/Jun/20 09:41

Description

When the data directory is set to EC, during the continuous load process, regionserver is found down ,to view the log:

regionserver.HeapMemoryManager: heapOccupancyPercent 0.91630864 is now below the heap occupancy alarm watermark (0.95)

regionserver.HeapMemoryManager: heapOccupancyPercent 0.969546 is above heap occupancy alarm watermark (0.95)

[JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 1254ms
GC pool 'G1 Young Generation' had collection(s): count=2 time=115ms
GC pool 'G1 Old Generation' had collection(s): count=1 time=1487ms

In particular, GC pauses occurred 283 times in half an hour.

The memory analysis found that most of the memory footprint was generated by HeepByteBuffer,and DFSStripedInputStream produces the HeepByteBuffer

See HDFS-14308，but after the fix, hbase did not return to normal.

Issue Links

• is related to: HDFS-14308 DFSStripedInputStream curStripeBuf is not freed by unbuffer()
• links to: GitHub Pull Request #1357
• links to: GitHub Pull Request #1404
• links to: GitHub Pull Request #1406
• links to: GitHub Pull Request #1415