Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-16458

LocatedFileStatusFetcher scans failing intermittently against S3 store

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.0
    • Fix Version/s: 3.3.0
    • Component/s: fs/s3
    • Labels:
      None
    • Environment:

      S3 + S3Guard

      Description

      Intermittent failure of LocatedFileStatusFetcher.getFileStatuses(), which is using globStatus to find files.

      I'd say "turn s3guard on" except this appears to be the case, and the dataset being read is
      over 1h old.

      Which means it is harder than I'd like to blame S3 for what would sound like an inconsistency

      We're hampered by the number of debug level statements in the globber code being approximately none; there's no debugging to turn on. All we know is that globFiles returns null without any explanation.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: