Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8256

Pluggable provider for node membership management

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.8.3, 3.0.2
    • None
    • resourcemanager
    • None

    Description

      Background

      HDFS-7541 introduced a pluggable provider framework for node membership management, which gives HDFS the flexibility to have different ways to manage node membership for different needs.

      org.apache.hadoop.hdfs.server.blockmanagement.HostConfigManager is the class which provides the abstraction. Currently, there are 2 implementations in the HDFS codebase:

      1) org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager which uses 2 config files which are defined by the properties dfs.hosts and dfs.hosts.exclude.

      2) org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager which uses a single JSON file defined by the property dfs.hosts.

      dfs.namenode.hosts.provider.classname is the property determining which implementation is used

      Problem

      YARN should be consistent with HDFS in terms of pluggable provider for node membership management. The absence of it makes YARN impossible to have other config sources, e.g., ZooKeeper, database, other config file formats, etc.

      Proposed solution

      org.apache.hadoop.yarn.server.resourcemanager.NodesListManager is the class for managing YARN node membership today. It uses HostsFileReader to read config files specified by the property yarn.resourcemanager.nodes.include-path for nodes to include and yarn.resourcemanager.nodes.nodes.exclude-path for nodes to exclude.

      The proposed solution is to

      1) introduce a new interface HostsConfigManager which provides the abstraction for node membership management. Update NodeListManager to depend on HostsConfigManager instead of HostsFileReader. Then create a wrapper class for HostsFileReader which implements the interface.

      2) introduce a new config property yarn.resourcemanager.hosts-config.manager.class for specifying the implementation class. Set the default value to the wrapper class of HostsFileReader for backward compatibility between new code and old config.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              functicons Dagang Wei
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - 24h
                  24h
                  Remaining:
                  Remaining Estimate - 24h
                  24h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified