Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
  3. HADOOP-13028

add low level counter metrics for S3A; use in read performance tests

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3, metrics
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      against S3 (and other object stores), opening connections can be expensive, closing connections may be expensive (a sign of a regression).

      S3A FS and individual input streams should have counters of the # of open/close/failure+reconnect operations, timers of how long things take. This can be used downstream to measure efficiency of the code (how often connections are being made), connection reliability, etc.

        Attachments

        1. HADOOP-13028-001.patch
          43 kB
          Steve Loughran
        2. HADOOP-13028-002.patch
          64 kB
          Steve Loughran
        3. HADOOP-13028-004.patch
          77 kB
          Steve Loughran
        4. HADOOP-13028-005.patch
          101 kB
          Steve Loughran
        5. HADOOP-13028-006.patch
          90 kB
          Steve Loughran
        6. HADOOP-13028-007.patch
          94 kB
          Steve Loughran
        7. HADOOP-13028-008.patch
          106 kB
          Steve Loughran
        8. HADOOP-13028-009.patch
          110 kB
          Steve Loughran
        9. HADOOP-13028-012.patch
          128 kB
          Steve Loughran
        10. HADOOP-13028-013.patch
          128 kB
          Steve Loughran
        11. HADOOP-13028-branch-2-008.patch
          106 kB
          Steve Loughran
        12. HADOOP-13028-branch-2-009.patch
          110 kB
          Steve Loughran
        13. HADOOP-13028-branch-2-010.patch
          128 kB
          Steve Loughran
        14. HADOOP-13028-branch-2-011.patch
          130 kB
          Steve Loughran
        15. HADOOP-13028-branch-2-012.patch
          130 kB
          Steve Loughran
        16. HADOOP-13028-branch-2-013.patch
          130 kB
          Steve Loughran
        17. org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
          94 kB
          Steve Loughran
        18. org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
          176 kB
          Steve Loughran

          Issue Links

          breaks

          Sub-task - The sub-task of the issue HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL.

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          contains

          Sub-task - The sub-task of the issue HADOOP-12844 Recover when S3A fails on IOException in read()

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-13047 S3a Forward seek in stream length to be configurable

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge is enabled

          • Major - Major loss of function.
          • Resolved
          is related to

          Sub-task - The sub-task of the issue HADOOP-12949 Add HTrace to the s3a connector

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-13065 Add a new interface for retrieving FS and FC Statistics

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-16241 S3AInputStream PositionReadable should perform ranged read on dedicated stream

          • Major - Major loss of function.
          • Open
          relates to

          Sub-task - The sub-task of the issue HADOOP-14862 Metrics for AdlFileSystem

          • Major - Major loss of function.
          • Open

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                1 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: