Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13204 Über-jira: S3a phase III: scale and tuning
  3. HADOOP-14738

Remove S3N and obsolete bits of S3A; rework docs

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 2.9.0, 3.0.0-beta1
    • 3.0.0-beta1
    • fs/s3
    • None
    • Incompatible change
    • Hide
      * The s3n:// client has been removed. Please upgrade to the s3a:// client.
      * The s3a's original output stream has been removed, the "fast" output stream is the sole option available. There is no need to explicitly enable this, and trying to disable it (fs.s3a.fast.upload=false) will have no effect.
      Show
      * The s3n:// client has been removed. Please upgrade to the s3a:// client. * The s3a's original output stream has been removed, the "fast" output stream is the sole option available. There is no need to explicitly enable this, and trying to disable it (fs.s3a.fast.upload=false) will have no effect.

    Description

      We are all happy with S3A; it's been stable since Hadoop 2.7 and high-perf since Hadoop 2.8

      It's now time to kill S3N off, remove the source, the tests, the transitive dependencies. This patch does that.

      It also removes the obsolete, original s3a output stream; the fast/block upload stream has been stable and is much more manageable and maintained (put differently: we don't ever look at the original S3A output stream, and tell people not to use it for performance reasons).

      As well as cutting the features, this patch updates the aws docs with

      • split out s3n migration page
      • split out troubleshooting page
      • rework of the "uploading data with s3a" section of index.md, as there's no need to discuss the slow upload except in the past tense...all that is needed is to list the buffering and thread tuning options of the block uploader.

      Attachments

        1. HADOOP-14738-006.patch
          316 kB
          Aaron Fabbri
        2. HADOOP-14738-005.patch
          316 kB
          Steve Loughran
        3. HADOOP-14738-004.patch
          317 kB
          Steve Loughran
        4. HADOOP-14738-003.patch
          254 kB
          Steve Loughran
        5. HADOOP-14738-002.patch
          113 kB
          Steve Loughran
        6. HADOOP-14739-001.patch
          111 kB
          Steve Loughran

        Issue Links

          contains

          Sub-task - The sub-task of the issue HADOOP-13894 s3a troubleshooting to cover the "JSON parse error" message

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14545 Uninitialized S3A instance NPEs on toString()

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-14834 Remove original S3A output stream

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          duplicates

          Sub-task - The sub-task of the issue HADOOP-14746 Cut S3AOutputStream

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          incorporates

          Sub-task - The sub-task of the issue HADOOP-14826 review S3 docs prior to 3.0-beta-1

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-14529 retire s3n

          • Major - Major loss of function.
          • Resolved
          supercedes

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-12981 Remove/deprecate s3native properties from S3NativeFileSystemConfigKeys and core-default.xml

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              stevel@apache.org Steve Loughran
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: