Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15619

Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      Features for S3Guard for Hadoop 3.3. Goal: take the experimental tag off

        Attachments

          Issue Links

          1.
          Add more s3guard metrics Sub-task Open Gabor Bota
          2.
          S3guard metadata stores to support millions of entries Sub-task Open Unassigned
          3.
          Improve DynamoDB schema update story Sub-task Open Sean Mackrory
          4.
          Ensure controls in-place to prevent clients with significant clock skews pruning aggressively Sub-task Open Unassigned
          5.
          S3Guard instrumentation to include cost of DynamoDB ops as metric Sub-task Open Unassigned
          6.
          increase performance of s3guard import command Sub-task Open Unassigned
          7.
          improvements to S3GuardTool destroy command Sub-task Open Gabor Bota
          8.
          Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily restrictive Sub-task Open Unassigned
          9.
          S3Guard: remove "experimental" from documentation Sub-task Open Steve Loughran
          10.
          Possible for modified configuration to leak into metadatastore in S3GuardTool Sub-task Open Unassigned
          11.
          hadoop fs -mkdir -p path-ending-with-slash/ fails with s3guard Sub-task Open Steve Loughran
          12.
          S3Guard to self update on directory listings of S3 Sub-task Open Gabor Bota
          13.
          reduce/tune read failure fault injection on inconsistent client Sub-task Open Unassigned
          14.
          S3guard mistakes root URI without / as non-absolute path Sub-task Open Unassigned
          15.
          S3guard: add inconsistency detection metrics Sub-task Open Gabor Bota
          16.
          ITestS3GuardListConsistency#testConsistentListAfterDelete fails when running with dynamo Sub-task Open Unassigned
          17.
          Improve throttling on S3Guard DDB batch retries Sub-task Open Unassigned
          18.
          tag S3GuardTool entry points as limitedPrivate("management-tools")/evolving Sub-task Open Steve Loughran
          19.
          getFileChecksum() needs to adopt S3Guard Sub-task In Progress lqjacklee
          20.
          Use lighter-weight alternatives to innerGetFileStatus where possible Sub-task Open Unassigned
          21.
          S3Guard: Optimize performance of handling OOB operations in non-authoritative mode Sub-task Open Unassigned
          22.
          transient failure of ITestS3GuardToolDynamoDB.testDestroyUnknownTable Sub-task Open Unassigned
          23.
          S3Guard: Retry failed read with backoff in Authoritative mode when file can be opened Sub-task Open Unassigned
          24.
          S3Guard: Remove LocalMetadataStore Sub-task Open Unassigned
          25.
          Add option for a prefix to put in front of every s3guard table Sub-task Open Unassigned
          26.
          DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry Sub-task Open Steve Loughran
          27.
          ZookeeperMetadataStore: Use Zookeeper as S3Guard backend store Sub-task Open Unassigned
          28.
          ITestS3GuardToolDynamoDB.testDynamoDBInitDestroyCycle fails if test bucket isn't on demand Sub-task Open Steve Loughran
          29.
          S3AFileSystem copyFile to propagate etag/version from getObjectMetadata to copy request Sub-task Open Unassigned
          30.
          Clock skew can cause S3Guard to think object metadata is out of date Sub-task Open Unassigned
          31.
          S3Guard Diff tool to list+ compare the etag and version fields Sub-task Open Unassigned
          32.
          S3a getFileStatus to update DDB if an S3 query returns etag/versionID Sub-task Open Unassigned
          33.
          mark DynamoDBMetadataStore.deleteTrackingValueMap as final Sub-task Patch Available kevin su
          34.
          Transient failure of ITestS3ARemoteFileChanged Sub-task Open Unassigned
          35.
          ITestS3GuardOutOfBandOperations.deleteAfterTombstoneExpiryOobCreate failure Sub-task Open Unassigned
          36.
          DynamoDBMetaStore deleteSubtree to delete leaf nodes first Sub-task Open Unassigned
          37.
          S3AFilesystem.delete to incrementally update s3guard with deletions Sub-task In Progress Steve Loughran
          38.
          Fix ITestS3GuardToolLocal#testInitNegativeRead test failure Sub-task Open Steve Loughran
          39.
          Intermittent failure of ITestS3GuardToolDynamoDB#testDynamoDBInitDestroyCycle Sub-task Open Unassigned
          40.
          S3Guard prune to only remove auth dir marker if files (not tombstones) are removed Sub-task Open Unassigned
          41.
          S3Guard ProgressiveRenameTracker to mark dest dir as authoritative on success Sub-task Open Unassigned
          42.
          Intermittent failure of ITestS3GuardConcurrentOps#testConcurrentTableCreations Sub-task Open Unassigned
          43.
          S3A to warn or fail if S3Guard is disabled Sub-task Open Unassigned
          44.
          S3Guard operations log has tombstone/PUT swapped Sub-task Open Unassigned
          45.
          S3A/S3Guard to avoid and handle FNFE eventual consistency better Sub-task Open Steve Loughran
          46.
          S3AFilesystem.initiateRename() can skip check on dest.parent status if src has same parent Sub-task Open Unassigned
          47.
          s3guard auth path checks only check against unqualified source path Sub-task Open Steve Loughran
          48.
          race condition in DDB table init and waiting threads Sub-task Open Unassigned

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: