[HDFS-5122] Support failover and retry in WebHdfsFileSystem for NN HA

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0-beta
• Fix Version/s: 2.3.0
• Component/s: ha, webhdfs
• Hadoop Flags: Reviewed
• Assignee: Haohui Mai
• Reporter: Arpit Gupta
• Created: 21/Aug/13 22:49
• Updated: 24/Feb/14 20:57
• Resolved: 18/Sep/13 21:35

Description

Bug reported by Arpit Gupta:

If the dfs.nameservices is set to arpit,

hdfs dfs -ls webhdfs://arpit/tmp

does not work. You have to provide the exact active namenode hostname. On an HA cluster using dfs client one should not need to provide the active nn hostname.

To fix this, we try to
1) let WebHdfsFileSystem support logical NN service name
2) add failover_and_retry functionality in WebHdfsFileSystem for NN HA

Issue Links

• depends upon: HDFS-5219 Add configuration keys for retry policy in WebHDFSFileSystem
• duplicates: HDFS-4299 WebHDFS Should Support HA Configuration
• is duplicated by: HDFS-5181 Fail-over support for HA cluster in WebHDFS
• is duplicated by: HDFS-5176 WebHDFS should support logical service names in URIs
• is related to: HDFS-5123 Hftp should support namenode logical service names in URI

Sub-Tasks

• webhdfs failover append to stand-by namenode fails: Open