Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16829 Über-jira: S3A Hadoop 3.3.1 features
  3. HADOOP-14585

Ensure controls in-place to prevent clients with significant clock skews pruning aggressively

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Won't Fix
    • 3.0.0-beta1
    • None
    • fs/s3
    • None

    Description

      From discussion on HADOOP-14499:

      bear in mind that we can't guarantee that the clocks of all clients are in sync; you don't want a client whose TZ setting is wrong to aggressively prune things. Had that happen in production with files in shared filestore. This is why ant -diagnostics checks time consistency with temp files...

      temp files work on a shared FS. AWS is actually somewhat sensitive to clocks: if your VM is too far out of time then auth actually fails, its ~+-15 minutes. There's some stuff in the Java SDK to actually calculate and adjust clock skew, presumably parsing the timestamp of a failure, calculating the difference and retrying. Which means that the field in SDKGlobalConfiguration could help identify the difference between local time and AWS time.

      Attachments

        Activity

          People

            Unassigned Unassigned
            mackrorysd Sean Mackrory
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: