Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6982

nntop: top­-like tool for name node users

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.7.0
    • Component/s: None
    • Labels:
      None

      Description

      In this jira we motivate the need for nntop, a tool that, similarly to what top does in Linux, gives the list of top users of the HDFS name node and gives insight about which users are sending majority of each traffic type to the name node. This information turns out to be the most critical when the name node is under pressure and the HDFS admin needs to know which user is hammering the name node and with what kind of requests. Here we present the design of nntop which has been in production at Twitter in the past 10 months. nntop proved to have low cpu overhead (< 2% in a cluster of 4K nodes), low memory footprint (less than a few MB), and quite efficient for the write path (only two hash lookup for updating a metric).

        Attachments

        1. nntop-design-v1.pdf
          412 kB
          Maysam Yabandeh
        2. HDFS-6982.v9.patch
          52 kB
          Maysam Yabandeh
        3. HDFS-6982.v8.patch
          49 kB
          Maysam Yabandeh
        4. HDFS-6982.v7.patch
          49 kB
          Maysam Yabandeh
        5. HDFS-6982.v6.patch
          48 kB
          Maysam Yabandeh
        6. HDFS-6982.v5.patch
          47 kB
          Maysam Yabandeh
        7. HDFS-6982.v4.patch
          48 kB
          Maysam Yabandeh
        8. HDFS-6982.v3.patch
          47 kB
          Maysam Yabandeh
        9. HDFS-6982.v2.patch
          48 kB
          Maysam Yabandeh
        10. HDFS-6982.v10.patch
          52 kB
          Maysam Yabandeh
        11. HDFS-6982.patch
          47 kB
          Maysam Yabandeh

          Issue Links

            Activity

              People

              • Assignee:
                maysamyabandeh Maysam Yabandeh
                Reporter:
                maysamyabandeh Maysam Yabandeh
              • Votes:
                0 Vote for this issue
                Watchers:
                27 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: