Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-13946

Non NameNode-HA properties still in hdfs-site.xml causing (at least) Balancer and ATS to fail

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.2, 2.2.0
    • Fix Version/s: 2.2.2
    • Component/s: ambari-server
    • Labels:
      None
    • Environment:

      CentOS6.7, HDP2.3-2950

      Description

      After enabling NameNode-HA, hdfs-site.xml does still contain non-HA properties, including

      • dfs.namenode.rpc-address
      • dfs.namenode.http-address
      • dfs.namenode.https-address

      This cause the balancer to fail with the following symptoms in Balancer:

      ...
      15/11/18 15:48:30 INFO balancer.Balancer: namenodes  = [hdfs://daplab2, hdfs://daplab-rt-11.fri.lan:8020]
      ...
      java.io.IOException: Another Balancer is running..  Exiting ...
      

      And ATS:

      _assert_valid
          self.target_status = self._get_file_status(target)
        File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 292, in _get_file_status
          list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
        File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 210, in run_command
          raise Fail(err_msg)
      resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://pvvsccmn1-brn1:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=403. 
      {
        "RemoteException": {
          "exception": "StandbyException", 
          "javaClassName": "org.apache.hadoop.ipc.StandbyException", 
          "message": "Operation category READ is not supported in state standby"
        }
      }
      

      These should be removed from the config.

      Steps to reproduce: after turning on NameNode HA, grep dfs.namenode.rpc-address|dfs.namenode.http-address /etc/hadoop/conf/hdfs-site.xml shouldn't return anything

        Attachments

        1. AMBARI-13946.patch
          0.8 kB
          Aleksandr Kovalenko
        2. AMBARI-13946_branch-2.2.patch
          0.8 kB
          Aleksandr Kovalenko

          Issue Links

            Activity

              People

              • Assignee:
                akovalenko Aleksandr Kovalenko
                Reporter:
                bperroud Benoit Perroud
              • Votes:
                1 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: