Description
Initially changes were made as part of this PR to handle idempotency including rename operation with the understanding that last modified time gets updated. But that assumption was wrong and the rename idempotency handling has since evolved.
For a job clean up, if the Manifest Committer in below Jira is used, then rename idempotency works using the previously fetched etag :
HADOOP-18163 hadoop-azure support for the Manifest Committer of MAPREDUCE-7341 - ASF JIRA (apache.org)
A part of the commit tracked under current Jira to handle DELETE idempotency is still relevant.
A means to handle idempotency between driver and backend inherently is being worked upon.
– Older notes
Currently when a PUT or POST operation timeouts and the server has already successfully executed the operation, there is no check in driver to see if the operation did succeed or not and just retries the same operation again. This can cause driver to through invalid user errors.
Sample scenario:
- Rename request times out. Though server has successfully executed the operation.
- Driver retries rename and get source not found error.
In the scenario, driver needs to check if rename is being retried and success if source if not found, but destination is present.
Attachments
Issue Links
- is related to
-
HADOOP-17132 ABFS: Fix For Idempotency code
-
- Resolved
-
- links to