[HDDS-3350] Ozone Retry Policy Improvements

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Ozone Client
• Labels: Triaged pull-request-available
• Target Version/s: 1.0.0
• Assignee: Lokesh Jain
• Reporter: Lokesh Jain
• Created: 06/Apr/20 14:57
• Updated: 21/Jul/20 02:36
• Resolved: 16/Jun/20 06:00

Description

Currently any ozone client request can spend a huge amount of time in retries and ozone client can retry its requests very aggressively. The waiting time can thus be very high before a client request fails. Further aggressive retries by ratis client used by ozone can bog down a ratis pipeline leader. The Jira aims to make changes to the current retry behavior in Ozone client. 

Issue Links

• blocks: HDDS-3994 Write object when met exception can be slower than before
• contains: HDDS-3136 retry timeout is large while writing key
• contains: HDDS-2702 Client failed to recover from ratis AlreadyClosedException exception
• contains: HDDS-2306 Fix TestWatchForCommit failure
• contains: HDDS-2963 Use RequestDependentRetry Policy along with ExceptionDependentRetry Policy in OzoneClient
• contains: HDDS-3163 write Key is hung when write delay is injected in datanode dir
• contains: HDDS-3268 CommitWatcher#watchForCommit does not timeout
• contains: HDDS-2917 Add a different retry policy for watch requests
• depends upon: RATIS-835 Include exception based attempt count in raft client request
• depends upon: RATIS-851 Raft Client should not change leader on ResourceUnavailableException
• depends upon: RATIS-853 Unordered Client request should not sleep when NotLeaderException provides leader information
• depends upon: RATIS-876 Introduce max timeout in RequestTypeDependentRetryPolicy
• depends upon: RATIS-882 Implement ExponentialBackoffRetry
• duplicates: HDDS-3228 Enforce Exponential backoff retry policy for raft client in cases of system overload
• links to: GitHub Pull Request #1005

Sub-Tasks

• retry timeout is large while writing key: Resolved