Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16830 Add public IOStatistics API
  3. HADOOP-17273

FSDataInput/Output Streams to automatically collect IOStatistics

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.3.1
    • Fix Version/s: None
    • Component/s: fs
    • Labels:
      None

      Description

      the fs input/output streams to automatically collect stream IO statistics even if the inner classes don't; count invocations and durations

      An issue here becomes "how would you aggregate with the inner stream, efficiently. Not sure there, except maybe

      • have some interface to access an updatable IOStatisticsStore which, if the stream implements, says "here is something you can update"
      • wrapper class updates that with counts, durations
      • we extend DynamicIOStatistics for each counter/gauge etc to have a factory which can add new entries on demand (new AtomicLong, etc)
      • so wrapper classes just update their stats, which updates existing stats or triggers off an on-demand creation of a new entry

      Not so informative in terms of low level details (HTTP/IPC requests and latency, errors, bytes discarded) but would give callers benefits of using the API for HDFS, Ozone, GCS

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: