Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-6100

DataNodeWebHdfsMethods does not failover in HA mode

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.4.0
    • Component/s: ha
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In DataNodeWebHdfsMethods, the code creates a DFSClient to connect to the NN, so that it can access the files in the cluster. DataNodeWebHdfsMethods relies on the address passed in the URL to locate the NN. This implementation has two problems:

      1. The DFSClient only knows about the current active NN, thus it does not support failover.
      2. The delegation token is based on the active NN, therefore the DFSClient will fail to authenticate of the standby NN in secure HA setup.

      Currently the parameter namenoderpcaddress in the URL stores the host-ip pair that corresponds to the active NN. To fix this bug, this jira proposes to store the name service id in the parameter in HA setup (yet the parameter stays the same in non-HA setup).

      1. HDFS-6100.001.patch
        37 kB
        Haohui Mai
      2. HDFS-6100.000.patch
        33 kB
        Haohui Mai

        Issue Links

          Activity

          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Chris Nauroth made changes -
          Link This issue breaks HDFS-6140 [ HDFS-6140 ]
          Jing Zhao made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.4.0 [ 12326143 ]
          Resolution Fixed [ 1 ]
          Haohui Mai made changes -
          Description In {{DataNodeWebHdfsMethods}}, the code creates a {{DFSClient}} to connect to the NN, so that it can access the files in the cluster. {{DataNodeWebHdfsMethods}} relies on the address passed in the URL to locate the NN. Currently the parameter is set by the NN and it is a host-ip pair, which does not support HA. In {{DataNodeWebHdfsMethods}}, the code creates a {{DFSClient}} to connect to the NN, so that it can access the files in the cluster. {{DataNodeWebHdfsMethods}} relies on the address passed in the URL to locate the NN. This implementation has two problems:

          # The {{DFSClient}} only knows about the current active NN, thus it does not support failover.
          # The delegation token is based on the active NN, therefore the {{DFSClient}} will fail to authenticate of the standby NN in secure HA setup.

          Currently the parameter {{namenoderpcaddress}} in the URL stores the host-ip pair that corresponds to the active NN. To fix this bug, this jira proposes to store the name service id in the parameter in HA setup (yet the parameter stays the same in non-HA setup).
          Haohui Mai made changes -
          Attachment HDFS-6100.001.patch [ 12635471 ]
          Haohui Mai made changes -
          Description While running slive with a webhdfs file system reducers fail as they keep trying to write to standby namenode. In {{DataNodeWebHdfsMethods}}, the code creates a {{DFSClient}} to connect to the NN, so that it can access the files in the cluster. {{DataNodeWebHdfsMethods}} relies on the address passed in the URL to locate the NN. Currently the parameter is set by the NN and it is a host-ip pair, which does not support HA.
          Haohui Mai made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Haohui Mai made changes -
          Attachment HDFS-6100.000.patch [ 12634593 ]
          Haohui Mai made changes -
          Field Original Value New Value
          Summary webhdfs filesystem does not failover in HA mode DataNodeWebHdfsMethods does not failover in HA mode
          Arpit Gupta created issue -

            People

            • Assignee:
              Haohui Mai
              Reporter:
              Arpit Gupta
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development