Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-18072 Über-JIRA: abfs phase III: Hadoop 3.3.x features & fixes
  3. HADOOP-18012

ABFS: Enable config controlled ETag check for Rename idempotency

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.2
    • None
    • fs/azure
    • None

    Description

      ABFS driver has a handling for rename idempotency which relies on LMT of the destination file to conclude if the rename was successful or not when source file is absent and if the rename request had entered retry loop.

      This handling is incorrect as LMT of the destination does not change on rename. 

      This Jira will track the change to undo the current implementation and add a new one where for an incoming rename operation, source file eTag is fetched first and then rename is done only if eTag matches for the source file.

      As this is going to be a costly operation given an extra HEAD request is added to each rename, this implementation will be guarded over a config and can enabled by customers who have workloads that do multiple renames. 

      Long term plan to handle rename idempotency without HEAD request is being discussed.

      Attachments

        Issue Links

          Activity

            People

              snvijaya Sneha Vijayarajan
              snvijaya Sneha Vijayarajan
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: