[HBASE-900] Regionserver memory leak causing OOME during relatively modest bulk importing

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.18.1, 0.19.0
• Fix Version/s: 0.19.0
• Assignee: Michael Stack
• Reporter: Jonathan Gray
• Created: 24/Sep/08 23:53
• Updated: 13/Sep/09 22:26
• Resolved: 19/Dec/08 19:03

Description

I have recreated this issue several times and it appears to have been introduced in 0.2.

During an import to a single table, memory usage of individual region servers grows w/o bounds and when set to the default 1GB it will eventually die with OOME. This has happened to me as well as Daniel Ploeg on the mailing list. In my case, I have 10 RS nodes and OOME happens w/ 1GB heap at only about 30-35 regions per RS. In previous versions, I have imported to several hundred regions per RS with default heap size.

I am able to get past this by increasing the max heap to 2GB. However, the appearance of this in newer versions leads me to believe there is now some kind of memory leak happening in the region servers during import.

Issue Links

• relates to: HBASE-1018 Regionservers should report detailed health to master; master should flag troubled regionservers in UI
• relates to: HBASE-1019 Master should reassign regions away from regionservers under heap stress