[AMBARI-13946] Non NameNode-HA properties still in hdfs-site.xml causing (at least) Balancer and ATS to fail

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.2, 2.2.0
• Fix Version/s: 2.2.2
• Component/s: ambari-server

Environment

CentOS6.7, HDP2.3-2950

• Assignee: Aleksandr Kovalenko
• Reporter: Benoit Perroud
• Created: 18/Nov/15 16:01
• Updated: 22/Mar/16 09:18
• Resolved: 25/Feb/16 17:56

Description

After enabling NameNode-HA, hdfs-site.xml does still contain non-HA properties, including

• dfs.namenode.rpc-address
• dfs.namenode.http-address
• dfs.namenode.https-address

This cause the balancer to fail with the following symptoms in Balancer:

...
15/11/18 15:48:30 INFO balancer.Balancer: namenodes  = [hdfs://daplab2, hdfs://daplab-rt-11.fri.lan:8020]
...
java.io.IOException: Another Balancer is running..  Exiting ...

And ATS:

_assert_valid
    self.target_status = self._get_file_status(target)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 292, in _get_file_status
    list_status = self.util.run_command(target, 'GETFILESTATUS', method='GET', ignore_status_codes=['404'], assertable_result=False)
  File "/usr/lib/python2.6/site-packages/resource_management/libraries/providers/hdfs_resource.py", line 210, in run_command
    raise Fail(err_msg)
resource_management.core.exceptions.Fail: Execution of 'curl -sS -L -w '%{http_code}' -X GET 'http://pvvsccmn1-brn1:50070/webhdfs/v1/ats/done?op=GETFILESTATUS&user.name=hdfs'' returned status_code=403. 
{
  "RemoteException": {
    "exception": "StandbyException", 
    "javaClassName": "org.apache.hadoop.ipc.StandbyException", 
    "message": "Operation category READ is not supported in state standby"
  }
}

These should be removed from the config.

Steps to reproduce: after turning on NameNode HA, grep dfs.namenode.rpc-address|dfs.namenode.http-address /etc/hadoop/conf/hdfs-site.xml shouldn't return anything

Issue Links

• is related to: HDFS-8897 Balancer should handle fs.defaultFS trailing slash in HA
• relates to: AMBARI-15506 Support for multiple Namservices in namenode_ha_utils.py