Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13055

Aggregate usage statistics from datanodes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We collect variety of statistics in DataNodes and expose them via JMX. Aggregating some of the high level statistics which we are already collecting in DataNodeMetrics (like bytesRead,bytesWritten etc) over a configurable time window will create a central repository accessible via JMX and UI. 

      Breaking this into 3 parts as suggested by [~arpitagarwal]

      1. Generate usage report on DN. (will handle in this jira)
      2. Pass usage report to NN periodically.
      3. Process report on NN and expose stats via JMX.
         

        Attachments

        1. HDFS-13055.001.patch
          58 kB
          Ajay Kumar
        2. HDFS-13055.002.patch
          59 kB
          Ajay Kumar
        3. HDFS-13055.003.patch
          59 kB
          Ajay Kumar
        4. HDFS-13055.004.patch
          59 kB
          Ajay Kumar
        5. HDFS-13055.005.patch
          61 kB
          Ajay Kumar
        6. HDFS-13055.006.patch
          61 kB
          Ajay Kumar
        7. HDFS-13055.007.patch
          18 kB
          Ajay Kumar
        8. HDFS-13055.008.patch
          18 kB
          Ajay Kumar
        9. HDFS-13055.009.patch
          18 kB
          Ajay Kumar
        10. HDFS-13055.010.patch
          18 kB
          Ajay Kumar

          Issue Links

            Activity

              People

              • Assignee:
                ajayydv Ajay Kumar
                Reporter:
                ajayydv Ajay Kumar
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: