Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15763 Über-JIRA: abfs phase II: Hadoop 3.3 features & fixes
  3. HADOOP-16340

ABFS driver continues to retry on IOException responses from REST operations

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.2.0, 3.3.0
    • Fix Version/s: 3.3.0
    • Component/s: fs/azure
    • Labels:
      None

      Description

      ABFS driver continues to retry (until retry count is exhausted) upon IOException responses from REST operations.  

      In the exception hander for IOExceptions at https://github.com/apache/hadoop/blob/65f60e56b082faf92e1cd3daee2569d8fc669c67/hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsRestOperation.java#L174-L197, there is no way exit out of the retry loop by re-throwing an exception unless one of the following conditions have been met:

      • The retry limit was hit
      • An HttpException was encountered

      From an org.apache.hadoop.fs.azurebfs.extensions.CustomTokenProviderAdaptee or org.apache.hadoop.fs.azurebfs.extensions.CustomDelegationTokenManager implementation, there is no way to create an org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.HttpException since the constructor is package private. 

      Either the exception handler needs to generic handle exceptions like java.nio.file.AccessDeniedException and java.io.FileNotFoundException, or the access to org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.HttpException needs to be set to that custom implementations can use it. 

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rlevas Robert Levas
                Reporter:
                rlevas Robert Levas
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: