Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
  3. HADOOP-13028

add low level counter metrics for S3A; use in read performance tests

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: fs/s3, metrics
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      against S3 (and other object stores), opening connections can be expensive, closing connections may be expensive (a sign of a regression).

      S3A FS and individual input streams should have counters of the # of open/close/failure+reconnect operations, timers of how long things take. This can be used downstream to measure efficiency of the code (how often connections are being made), connection reliability, etc.

        Attachments

        1. HADOOP-13028-013.patch
          128 kB
          Steve Loughran
        2. HADOOP-13028-branch-2-013.patch
          130 kB
          Steve Loughran
        3. HADOOP-13028-012.patch
          128 kB
          Steve Loughran
        4. HADOOP-13028-branch-2-012.patch
          130 kB
          Steve Loughran
        5. HADOOP-13028-branch-2-011.patch
          130 kB
          Steve Loughran
        6. HADOOP-13028-branch-2-010.patch
          128 kB
          Steve Loughran
        7. HADOOP-13028-branch-2-009.patch
          110 kB
          Steve Loughran
        8. HADOOP-13028-009.patch
          110 kB
          Steve Loughran
        9. HADOOP-13028-branch-2-008.patch
          106 kB
          Steve Loughran
        10. HADOOP-13028-008.patch
          106 kB
          Steve Loughran
        11. HADOOP-13028-007.patch
          94 kB
          Steve Loughran
        12. HADOOP-13028-006.patch
          90 kB
          Steve Loughran
        13. HADOOP-13028-005.patch
          101 kB
          Steve Loughran
        14. HADOOP-13028-004.patch
          77 kB
          Steve Loughran
        15. org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
          94 kB
          Steve Loughran
        16. HADOOP-13028-002.patch
          64 kB
          Steve Loughran
        17. org.apache.hadoop.fs.s3a.scale.TestS3AInputStreamPerformance-output.txt
          176 kB
          Steve Loughran
        18. HADOOP-13028-001.patch
          43 kB
          Steve Loughran

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                1 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: