[AMBARI-17901] Make HDFS operations resilient to namenode safemode

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: ambari-server
• Assignee: Magyari Sandor Szilard
• Reporter: Magyari Sandor Szilard
• Created: 26/Jul/16 14:10
• Updated: 22/Dec/17 14:12
• Resolved: 22/Dec/17 14:12

Description

HdfsResourceJar and HdfsResourceWebHDFS (WebHDFSUtil) are the classes that carry out the HDFS operations. All retry able operations (e.g. SETPERMISSION) should be guarded with retry logic that would retry the operation until a given timeout before giving up and bailing out.

To determine which HDFS operations are retry able might be as easy as just looking the returned status/error code or the type of the exception (e.g. "RetriableException") though this needs to be verified if it's consistent with both the webhdfs and hdfsresource jar.

This problem came up in https://issues.apache.org/jira/browse/AMBARI-17182 when starting all services after Enabling HA.
Retry count and timeout should be clarified, as sometimes it may take a long time for namenode to exit safemode.

Issue Links

• relates to: AMBARI-17182 App timeline Server start fails on enabling HA because namenode is in safemode