Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11694

Über-jira: S3a phase II: robustness, scale and performance

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.0
    • Fix Version/s: 2.8.0
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      HADOOP-11571 covered the core s3a bugs surfacing in Hadoop-2.6 & other enhancements to improve S3 (performance, proxy, custom endpoints)

      This JIRA covers post-2.7 issues and enhancements.

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          stevel@apache.org Steve Loughran added a comment -

          Closing as fixed now HADOOP-14028 is done and Hadoop 2.8 is shipping. Happy now

          Show
          stevel@apache.org Steve Loughran added a comment - Closing as fixed now HADOOP-14028 is done and Hadoop 2.8 is shipping. Happy now
          Hide
          stevel@apache.org Steve Loughran added a comment -

          assuming this covers all for 2.8, there are some late-breaking bugs coming up..reopening

          Show
          stevel@apache.org Steve Loughran added a comment - assuming this covers all for 2.8, there are some late-breaking bugs coming up..reopening
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Note that I've not quite closed it, we have that thread pool one for me to look at...

          Show
          stevel@apache.org Steve Loughran added a comment - Note that I've not quite closed it, we have that thread pool one for me to look at...
          Hide
          stevel@apache.org Steve Loughran added a comment -

          I am declaring this JIRA done: the key stuff for Hadoop 2.8 is all in!

          I've moved some outstanding stuff to the Phase III JIRA, but the big bits: high performance reads, writes directory listings, they are all in.

          There's one thing we may actually want to pull up. HADOOP-13336 -in which you can specify per-bucket config. We need this for s3guard, but it's also useful when trying to do work between buckets with different logins. Putting it in 2.8 will get it out the door faster

          Show
          stevel@apache.org Steve Loughran added a comment - I am declaring this JIRA done: the key stuff for Hadoop 2.8 is all in! I've moved some outstanding stuff to the Phase III JIRA, but the big bits: high performance reads, writes directory listings, they are all in. There's one thing we may actually want to pull up. HADOOP-13336 -in which you can specify per-bucket config. We need this for s3guard, but it's also useful when trying to do work between buckets with different logins. Putting it in 2.8 will get it out the door faster
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Add HADOOP-12807 for env var pickup

          Show
          stevel@apache.org Steve Loughran added a comment - Add HADOOP-12807 for env var pickup
          Hide
          stevel@apache.org Steve Loughran added a comment -

          needs a test, HADOOP-13166 to stat the root path

          Show
          stevel@apache.org Steve Loughran added a comment - needs a test, HADOOP-13166 to stat the root path
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Link to HADOOP-12020; support for reduced-replication

          Show
          stevel@apache.org Steve Loughran added a comment - Link to HADOOP-12020 ; support for reduced-replication
          Hide
          stevel@apache.org Steve Loughran added a comment -

          As with Phase I, Jenkins doesn't run these tests by default. All submitters are expected to have AWS test setup, or their own local S3-compatible endpoints.

          Show
          stevel@apache.org Steve Loughran added a comment - As with Phase I, Jenkins doesn't run these tests by default. All submitters are expected to have AWS test setup, or their own local S3-compatible endpoints.

            People

            • Assignee:
              stevel@apache.org Steve Loughran
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              41 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 50h
                50h
                Remaining:
                Remaining Estimate - 50h
                50h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development