Hadoop Common
  1. Hadoop Common
  2. HADOOP-7359

Pluggable interface for cluster membership

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently Hadoop uses local files to determine cluster membership. With HDFS for example, dfs.hosts and dfs.hosts.exclude are used.

      To enable tighter integrations cluster membership should be an interface, with the current file-based functionality provided as the default implementation. The common case would be no functional change, however, sites could plug an alternative implementation in, such as pulling the machine lists from a machine database.

      DETAILS:

      Two machine lists, includes and excludes, are used to define cluster membership and state. HostsFileReader currently handles reading these lists from files, who's names are passed in by FSNamesystem for HDFS and JobTracker for MR.

      The proposed change is adding a HostsReader interface to common, and changing HostsFileReader to an abstract class that functions the same as today.

      Two new classes, DFSHostsFileReader and MRHostsFileReader, extend HostsFileReader and simply pass the appropriate file names in. These new classes are needed because config key names live outside common.

      Two new conf keys, defaulting to the file-based readers, would be added to choose a different hosts reader: dfs.namenode.hosts.reader.class mapreduce.jobtracker.hosts.reader.class

      Comments/suggestions? I have most of this written already but would love some feedback on the general idea before posting the diff.

      1. HADOOP-7359.diff
        59 kB
        Travis Crawford

        Issue Links

          Activity

          Travis Crawford created issue -
          Travis Crawford made changes -
          Field Original Value New Value
          Link This issue blocks MAPREDUCE-2567 [ MAPREDUCE-2567 ]
          Travis Crawford made changes -
          Link This issue blocks HDFS-2035 [ HDFS-2035 ]
          Travis Crawford made changes -
          Attachment HADOOP-7359.diff [ 12481486 ]
          Travis Crawford made changes -
          Attachment HADOOP-7359.diff [ 12481486 ]
          Travis Crawford made changes -
          Attachment HADOOP-7359.diff [ 12485065 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Travis Crawford
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:

                Development