Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10839

Add unregisterSource() to MetricsSystem API

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.1
    • Fix Version/s: 2.6.0
    • Component/s: metrics
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The MetricsSystem abstract class has added a new abstract method, unregisterSource, for unregistering a previously registered metrics source. Custom subclasses of MetricsSystem must be updated to provide an implementation of this method.

      Description

      Currently the MetrisSystem API has register() method to register a MetricsSource but doesn't have unregister() method. This means once a MetricsSource is registered with the MetricsSystem, it will be there forever until the MetricsSystem is shut down. This in some cases can cause Java OutOfMemoryError.

      One such case is in file system metrics implementation. The new AbstractFileSystem/FileContext framework does not implement a cache so every file system access can lead to the creation of a NativeFileSystem instance. (refer to HADOOP-6356). And all these NativeFileSystem needs to share the same instance of MetricsSystemImpl, which means we cannot shut down MetricsSystem to clean up all the MetricsSources that has been registered but no longer active. Over time the MetricsSource instance accumulates and eventually we saw OutOfMemoryError.

        Attachments

        1. HADOOP-10839.2.patch
          3 kB
          Chris Nauroth
        2. HADOOP-10839.patch
          3 kB
          shanyu zhao

          Issue Links

            Activity

              People

              • Assignee:
                shanyu shanyu zhao
                Reporter:
                shanyu shanyu zhao
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: