Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15707

NNTop counts don't add up as expected

    XMLWordPrintableJSON

    Details

      Description

      When we queried https://host-name:port/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState earlier today, we received surprising results. Below is a subset of the data. The notable thing here is that the count for opType='*' from user user1 is 66652, but the sum of counts for other optype values by user1 is actually larger: 77253. I'd expect that opType='*' to always be greater than or equal to the sum for individual operation types.

      {
        "windows": [
          {
            "windowLenMs": 300000,
            "ops": [
              {
                "totalCount": 24158,
                "opType": "rpc.complete",
                "topUsers": [{ "count": 2944, "user": "user1" }]
              },
              {
                "totalCount": 15921,
                "opType": "rpc.rename",
                "topUsers": [{ "count": 2891, "user": "user1" }]
              },
              {
                "totalCount": 3015834,
                "opType": "*",
                "topUsers": [{ "count": 66652, "user": "user1" }]
              },
              {
                "totalCount": 2086,
                "opType": "rpc.abandonBlock",
                "topUsers": [{ "count": 603, "user": "user1" }]
              },
              {
                "totalCount": 30258,
                "opType": "rpc.addBlock",
                "topUsers": [{ "count": 3182, "user": "user1" }]
              },
              {
                "totalCount": 101440,
                "opType": "rpc.getServerDefaults",
                "topUsers": [{ "count": 3521, "user": "user1" }]
              },
              {
                "totalCount": 25258,
                "opType": "rpc.create",
                "topUsers": [{ "count": 1864, "user": "user1" }]
              },
              {
                "totalCount": 1377563,
                "opType": "rpc.getFileInfo",
                "topUsers": [{ "count": 56541, "user": "user1" }]
              },
              {
                "totalCount": 60836,
                "opType": "rpc.renewLease",
                "topUsers": [{ "count": 3783, "user": "user1" }]
              },
              {
                "totalCount": 182212,
                "opType": "rpc.getListing",
                "topUsers": [{ "count": 1848, "user": "user1" }]
              },
              {
                "totalCount": 380,
                "opType": "rpc.updateBlockForPipeline",
                "topUsers": [{ "count": 58, "user": "user1" }]
              },
              {
                "totalCount": 215,
                "opType": "rpc.updatePipeline",
                "topUsers": [{ "count": 18, "user": "user1" }]
              }
            ]
          }
        ],
        "timestamp": "2017-07-11T17:20:03+0000"
      }
      

      The full TopUserOpCounts results from the NNTop endpoint results are attached to this JIRA as nntop.json. The full endpoint output is attached as endpoint.json.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ahussein Ahmed Hussein
                Reporter:
                ahussein Ahmed Hussein
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 50m
                  50m