[HADOOP-10840] Fix OutOfMemoryError caused by metrics system in Azure File System

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.1
• Fix Version/s: 2.7.0
• Component/s: metrics
• Target Version/s: 2.7.0
• Hadoop Flags: Reviewed
• Assignee: shanyu zhao
• Reporter: shanyu zhao
• Created: 15/Jul/14 20:59
• Updated: 10/Apr/15 20:04
• Resolved: 30/Oct/14 00:51

Description

In Hadoop 2.x the Hadoop File System framework changed and no cache is implemented (refer to HADOOP-6356). This means for every WASB access, a new NativeAzureFileSystem is created, along which a Metrics source created and added to MetricsSystemImpl. Over time the sources accumulated, eating memory and causing Java OutOfMemoryError.

The fix is to utilize the unregisterSource() method added to MetricsSystem in HADOOP-10839.

Issue Links

• relates to: HADOOP-9629 Support Windows Azure Storage - Blob as a file system in Hadoop
• requires: HADOOP-10839 Add unregisterSource() to MetricsSystem API