Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619 Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features
  3. HADOOP-16280

S3Guard: Retry failed read with backoff in Authoritative mode when file can be opened

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      When using S3Guard in authoritative mode a file can be reported from AWS S3 that's missing like it is described in the following exception:

      java.io.FileNotFoundException: re-open s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0 at 0 on 
      s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0: 
      com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not exist. (Service: Amazon S3; Status Code: 404; Error 
      Code: NoSuchKey; Request ID: E1FF9EA9B5DBBD7E; S3 Extended Request ID: 
      NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=), S3 Extended Request ID: 
      NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=:NoSuchKey
      

      But the metadata in S3Guard (e.g dynamo db) is there, so it can be opened. The operation will not fail when it's opened, it will fail when we try to read it, so the call

      FSDataInputStream is = guardedFs.open(testFilePath);

      won't fail, but the next call

      byte[] firstRead = new byte[text.length()];
      is.read(firstRead, 0, firstRead.length);
      

      will fail with the exception message like what's above.

      Once Authoritative mode is on, we assume that there is no out of band operation, so the file will appear eventually. We should re-try in this case.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                gabor.bota Gabor Bota
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: