Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16829 Über-jira: S3A Hadoop 3.4 features
  3. HADOOP-14585

Ensure controls in-place to prevent clients with significant clock skews pruning aggressively

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      From discussion on HADOOP-14499:

      bear in mind that we can't guarantee that the clocks of all clients are in sync; you don't want a client whose TZ setting is wrong to aggressively prune things. Had that happen in production with files in shared filestore. This is why ant -diagnostics checks time consistency with temp files...

      temp files work on a shared FS. AWS is actually somewhat sensitive to clocks: if your VM is too far out of time then auth actually fails, its ~+-15 minutes. There's some stuff in the Java SDK to actually calculate and adjust clock skew, presumably parsing the timestamp of a failure, calculating the difference and retrying. Which means that the field in SDKGlobalConfiguration could help identify the difference between local time and AWS time.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mackrorysd Sean Mackrory
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: