Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13065

Add a new interface for retrieving FS and FC Statistics

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • fs
    • None

    Description

      Currently FileSystem.Statistics exposes the following statistics:
      BytesRead
      BytesWritten
      ReadOps
      LargeReadOps
      WriteOps

      These are in-turn exposed as job counters by MapReduce and other frameworks. There is logic within DfsClient to map operations to these counters that can be confusing, for instance, mkdirs counts as a writeOp.

      Proposed enhancement:
      Add a statistic for each DfsClient operation including create, append, createSymlink, delete, exists, mkdirs, rename and expose them as new properties on the Statistics object. The operation-specific counters can be used for analyzing the load imposed by a particular job on HDFS.
      For example, we can use them to identify jobs that end up creating a large number of files.

      Once this information is available in the Statistics object, the app frameworks like MapReduce can expose them as additional counters to be aggregated and recorded as part of job summary.

      Attachments

        1. HADOOP-13065.008.patch
          73 kB
          Mingliang Liu
        2. HADOOP-13065.009.patch
          76 kB
          Mingliang Liu
        3. HADOOP-13065.010.patch
          77 kB
          Mingliang Liu
        4. HADOOP-13065.011.patch
          79 kB
          Mingliang Liu
        5. HADOOP-13065.012.patch
          79 kB
          Mingliang Liu
        6. HADOOP-13065.013.patch
          79 kB
          Mingliang Liu
        7. HADOOP-13065-007.patch
          20 kB
          Colin McCabe
        8. HDFS-10175.000.patch
          51 kB
          Mingliang Liu
        9. HDFS-10175.001.patch
          48 kB
          Mingliang Liu
        10. HDFS-10175.002.patch
          47 kB
          Mingliang Liu
        11. HDFS-10175.003.patch
          49 kB
          Mingliang Liu
        12. HDFS-10175.004.patch
          49 kB
          Mingliang Liu
        13. HDFS-10175.005.patch
          53 kB
          Mingliang Liu
        14. HDFS-10175.006.patch
          8 kB
          Colin McCabe
        15. TestStatisticsOverhead.java
          3 kB
          Colin McCabe

        Issue Links

          breaks

          Bug - A problem which impairs or prevents the functions of the product. HDFS-10418 NPE in TestDistributedFileSystem.testDFSCloseOrdering

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue HADOOP-13028 add low level counter metrics for S3A; use in read performance tests

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-13171 Add StorageStatistics to S3A; instrument some more operations

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Sub-task - The sub-task of the issue HADOOP-15124 Slow FileSystem.Statistics counters implementation

          • Major - Major loss of function.
          • Patch Available

          Bug - A problem which impairs or prevents the functions of the product. TEZ-3331 Add operation specific HDFS counters for Tez UI

          • Major - Major loss of function.
          • Patch Available
          There are no Sub-Tasks for this issue.

          Activity

            People

              liuml07 Mingliang Liu
              venkateshrin Ram Venkatesh
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: