Hadoop Common
  1. Hadoop Common
  2. HADOOP-8193

Refactor FailoverController/HAAdmin code to add an abstract class for "target" services

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.3, 0.24.0
    • Fix Version/s: 2.0.0-alpha
    • Component/s: ha
    • Labels:
      None

      Description

      In working at HADOOP-8077, HDFS-3084, and HDFS-3072, I ran into various difficulties which are an artifact of the current design. A few of these:

      • the service name is "resolved" from the logical name (eg ns1.nn1) to an IP address at the outer layer of DFSHAAdmin
        • this means it's difficult to provide the logical name "ns1.nn1" to fence scripts (HDFS-3084)
        • this means it's difficult to configure fencing method per-namespace (since the FailoverController doesn't know what the namespace is) (HADOOP-8077)
      • the configuration for HA HDFS is weirdly split between core-site and hdfs-site, even though most users see this as an HDFS feature. For example, users expect to configure NN fencing configurations in hdfs-site, and expect the keys to have a dfs.* prefix
      • proxies are constructed at the outer layer of the admin commands. This means it's impossible for the inner layers (eg FailoverController.failover) to re-construct proxies with different timeouts (HDFS-3072)

      The proposed refactor is to add a new interface (tentatively named HAServiceTarget) which refers to target for one of the admin commands. An instance of this class is responsible for creating proxies, creating fencers, mapping back to a logical name, etc. The HDFS implementation of this class can then provide different results based on the particular nameservice, can use HDFS-specific configuration prefixes, etc. Using this class as the argument for fencing methods also makes the API more evolvable in the future, since we can add new getters to HAServiceTarget (whereas the current InetSocketAddress is quite limiting)

      1. hdfs-8193-fix-npe.txt
        6 kB
        Todd Lipcon
      2. hadoop-8193.txt
        68 kB
        Todd Lipcon
      3. hadoop-8193.txt
        68 kB
        Todd Lipcon
      4. hadoop-8193.txt
        54 kB
        Todd Lipcon

        Issue Links

          Activity

          Todd Lipcon created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Link This issue incorporates HADOOP-8077 [ HADOOP-8077 ]
          Todd Lipcon made changes -
          Link This issue incorporates HDFS-3084 [ HDFS-3084 ]
          Todd Lipcon made changes -
          Link This issue incorporates HDFS-3072 [ HDFS-3072 ]
          Todd Lipcon made changes -
          Attachment hadoop-8193.txt [ 12519133 ]
          Todd Lipcon made changes -
          Attachment hadoop-8193.txt [ 12519566 ]
          Todd Lipcon made changes -
          Link This issue blocks HADOOP-8204 [ HADOOP-8204 ]
          Todd Lipcon made changes -
          Attachment hadoop-8193.txt [ 12519776 ]
          Todd Lipcon made changes -
          Link This issue is depended upon by HDFS-3072 [ HDFS-3072 ]
          Todd Lipcon made changes -
          Link This issue is depended upon by HADOOP-8077 [ HADOOP-8077 ]
          Todd Lipcon made changes -
          Link This issue is depended upon by HDFS-3084 [ HDFS-3084 ]
          Todd Lipcon made changes -
          Link This issue incorporates HDFS-3072 [ HDFS-3072 ]
          Todd Lipcon made changes -
          Link This issue incorporates HDFS-3084 [ HDFS-3084 ]
          Todd Lipcon made changes -
          Link This issue incorporates HADOOP-8077 [ HADOOP-8077 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.23.3 [ 12320059 ]
          Fix Version/s 0.24.0 [ 12317652 ]
          Resolution Fixed [ 1 ]
          Todd Lipcon made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Todd Lipcon made changes -
          Attachment hdfs-8193-fix-npe.txt [ 12519881 ]
          Todd Lipcon made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.0 [ 12320352 ]
          Fix Version/s 0.24.0 [ 12317652 ]
          Fix Version/s 0.23.3 [ 12320059 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development