[HDFS-15707] NNTop counts don't add up as expected

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.2, 3.3.1, 3.4.0, 3.2.3
• Component/s: hdfs, metrics, namenode
• Labels: pull-request-available
• Assignee: Ahmed Hussein
• Reporter: Ahmed Hussein
• Created: 03/Dec/20 19:56
• Updated: 10/Jun/21 07:44
• Resolved: 07/Dec/20 22:05

Description

When we queried https://host-name:port/jmx?qry=Hadoop:service=NameNode,name=FSNamesystemState earlier today, we received surprising results. Below is a subset of the data. The notable thing here is that the count for opType='*' from user user1 is 66652, but the sum of counts for other optype values by user1 is actually larger: 77253. I'd expect that opType='*' to always be greater than or equal to the sum for individual operation types.

{
  "windows": [
    {
      "windowLenMs": 300000,
      "ops": [
        {
          "totalCount": 24158,
          "opType": "rpc.complete",
          "topUsers": [{ "count": 2944, "user": "user1" }]
        },
        {
          "totalCount": 15921,
          "opType": "rpc.rename",
          "topUsers": [{ "count": 2891, "user": "user1" }]
        },
        {
          "totalCount": 3015834,
          "opType": "*",
          "topUsers": [{ "count": 66652, "user": "user1" }]
        },
        {
          "totalCount": 2086,
          "opType": "rpc.abandonBlock",
          "topUsers": [{ "count": 603, "user": "user1" }]
        },
        {
          "totalCount": 30258,
          "opType": "rpc.addBlock",
          "topUsers": [{ "count": 3182, "user": "user1" }]
        },
        {
          "totalCount": 101440,
          "opType": "rpc.getServerDefaults",
          "topUsers": [{ "count": 3521, "user": "user1" }]
        },
        {
          "totalCount": 25258,
          "opType": "rpc.create",
          "topUsers": [{ "count": 1864, "user": "user1" }]
        },
        {
          "totalCount": 1377563,
          "opType": "rpc.getFileInfo",
          "topUsers": [{ "count": 56541, "user": "user1" }]
        },
        {
          "totalCount": 60836,
          "opType": "rpc.renewLease",
          "topUsers": [{ "count": 3783, "user": "user1" }]
        },
        {
          "totalCount": 182212,
          "opType": "rpc.getListing",
          "topUsers": [{ "count": 1848, "user": "user1" }]
        },
        {
          "totalCount": 380,
          "opType": "rpc.updateBlockForPipeline",
          "topUsers": [{ "count": 58, "user": "user1" }]
        },
        {
          "totalCount": 215,
          "opType": "rpc.updatePipeline",
          "topUsers": [{ "count": 18, "user": "user1" }]
        }
      ]
    }
  ],
  "timestamp": "2017-07-11T17:20:03+0000"
}

The full TopUserOpCounts results from the NNTop endpoint results are attached to this JIRA as nntop.json. The full endpoint output is attached as endpoint.json.

Issue Links

• links to: GitHub Pull Request #2516