[YARN-8256] Pluggable provider for node membership management

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.8.3, 3.0.2
• Component/s: resourcemanager
• Target Version/s: 3.4.0
• Assignee: Unassigned
• Reporter: Dagang Wei
• Created: 07/May/18 21:43
• Updated: 09/Apr/20 18:41

Description

Background

HDFS-7541 introduced a pluggable provider framework for node membership management, which gives HDFS the flexibility to have different ways to manage node membership for different needs.

org.apache.hadoop.hdfs.server.blockmanagement.HostConfigManager is the class which provides the abstraction. Currently, there are 2 implementations in the HDFS codebase:

1) org.apache.hadoop.hdfs.server.blockmanagement.HostFileManager which uses 2 config files which are defined by the properties dfs.hosts and dfs.hosts.exclude.

2) org.apache.hadoop.hdfs.server.blockmanagement.CombinedHostFileManager which uses a single JSON file defined by the property dfs.hosts.

dfs.namenode.hosts.provider.classname is the property determining which implementation is used

Problem

YARN should be consistent with HDFS in terms of pluggable provider for node membership management. The absence of it makes YARN impossible to have other config sources, e.g., ZooKeeper, database, other config file formats, etc.

Proposed solution

org.apache.hadoop.yarn.server.resourcemanager.NodesListManager is the class for managing YARN node membership today. It uses HostsFileReader to read config files specified by the property yarn.resourcemanager.nodes.include-path for nodes to include and yarn.resourcemanager.nodes.nodes.exclude-path for nodes to exclude.

The proposed solution is to

1) introduce a new interface HostsConfigManager which provides the abstraction for node membership management. Update NodeListManager to depend on HostsConfigManager instead of HostsFileReader. Then create a wrapper class for HostsFileReader which implements the interface.

2) introduce a new config property yarn.resourcemanager.hosts-config.manager.class for specifying the implementation class. Set the default value to the wrapper class of HostsFileReader for backward compatibility between new code and old config.

Issue Links

• is related to: HDFS-7541 Upgrade Domains in HDFS
• links to: GitHub Pull Request #384
• links to: GitHub Pull Request #384