Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14084

Need for more stats in DFSClient

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Made changes to fix the issue

      Description

      The usage of HDFS has changed from being used as a map-reduce filesystem, now it's becoming more of like a general purpose filesystem. In most of the cases there are issues with the Namenode so we have metrics to know the workload or stress on Namenode.

      However, there is a need to have more statistics collected for different operations/RPCs in DFSClient to know which RPC operations are taking longer time or to know what is the frequency of the operation.These statistics can be exposed to the users of DFS Client and they can periodically log or do some sort of flow control if the response is slow. This will also help to isolate HDFS issue in a mixed environment where on a node say we have Spark, HBase and Impala running together. We can check the throughput of different operation across client and isolate the problem caused because of noisy neighbor or network congestion or shared JVM.

      We have dealt with several problems from the field for which there is no conclusive evidence as to what caused the problem. If we had metrics or stats in DFSClient we would be better equipped to solve such complex problems.

      List of jiras for reference:
      -------------------------
      HADOOP-15538 HADOOP-15530 ( client side deadlock)

        Attachments

        1. HDFS-14084.001.patch
          10 kB
          Pranay Singh
        2. HDFS-14084.002.patch
          5 kB
          Pranay Singh
        3. HDFS-14084.003.patch
          10 kB
          Pranay Singh
        4. HDFS-14084.004.patch
          10 kB
          Pranay Singh
        5. HDFS-14084.005.patch
          11 kB
          Pranay Singh
        6. HDFS-14084.006.patch
          11 kB
          Pranay Singh
        7. HDFS-14084.007.patch
          11 kB
          Pranay Singh
        8. HDFS-14084.008.patch
          11 kB
          Pranay Singh
        9. HDFS-14084.009.patch
          10 kB
          Pranay Singh
        10. HDFS-14084.010.patch
          10 kB
          Pranay Singh
        11. HDFS-14084.011.patch
          10 kB
          Pranay Singh
        12. HDFS-14084.012.patch
          11 kB
          Pranay Singh
        13. HDFS-14084.013.patch
          12 kB
          Pranay Singh
        14. HDFS-14084.014.patch
          2 kB
          Pranay Singh
        15. HDFS-14084.015.patch
          10 kB
          Pranay Singh
        16. HDFS-14084.016.patch
          10 kB
          Pranay Singh
        17. HDFS-14084.017.patch
          11 kB
          Pranay Singh
        18. HDFS-14084.018.patch
          11 kB
          Pranay Singh

          Issue Links

            Activity

              People

              • Assignee:
                xkrogen Erik Krogen
                Reporter:
                pranay_singh Pranay Singh
              • Votes:
                0 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated: