Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15620 Über-jira: S3A phase VI: Hadoop 3.3 features
  3. HADOOP-16379

S3AInputStream#unbuffer should merge input stream stats into fs-wide stats

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.0
    • 3.3.0
    • fs/s3
    • None

    Description

      HADOOP-14747 added support for CanUnbuffer to S3A. One issue for applications that rely on the CanUnbuffer functionality, is that the current S3A code only merges input stream stats when the stream is closed. Applications using CanUnbuffer might not close streams for a long time, so any stats reported by input streams don't get reported in the filesystem-wide metrics.

      This JIRA proposes merging input stream statistics when S3AInputStream#unbuffer is called.

      Attachments

        Issue Links

          is required by

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-8544 Expose additional S3A / S3Guard metrics

          • Major - Major loss of function.
          • Open
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-28091 Extend Spark metrics system with user-defined metrics using executor plugins

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link GitHub Pull Request #983

          Activity

            People

              stakiar Sahil Takiar
              stakiar Sahil Takiar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: