Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7176

Impala fails to cancel a fragment in test_insert_mem_limit

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.13.0
    • Fix Version/s: Impala 3.3.0
    • Component/s: Backend
    • Labels:

      Description

      query_test/test_insert.py::TestInsertQueries::test_insert_mem_limit failed. A query failed but the num-fragments-in-flight doesn't reach 0 in 60 seconds. The offending fragment seems to have a hard time closing HDFS FDs:

      I0613 12:43:23.338606 20915 runtime-state.cc:175] Error from query 874a1bc05259727f:d244cabe00000000: Failed to close HDFS file: hdfs://localhost:20500/test-warehouse/functional_parquet.db/alltypesinsert/_impala_insert_staging/874a1
      bc05259727f_d244cabe00000000/.874a1bc05259727f-d244cabe00000007_891753484_dir/year=2010/month=5522/874a1bc05259727f-d244cabe00000007_10566929_data.0.parq
      Error(255): Unknown error 255
      Root cause: RemoteException: No lease on /test-warehouse/functional_parquet.db/alltypesinsert/_impala_insert_staging/874a1bc05259727f_d244cabe00000000/.874a1bc05259727f-d244cabe00000007_891753484_dir/year=2010/month=5522/874a1bc0525
      9727f-d244cabe00000007_10566929_data.0.parq (inode 112838): File does not exist. [Lease.  Holder: DFSClient_NONMAPREDUCE_-1969506178_1, pending creates: 54]
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3817)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3618)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3474)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:694)
        at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:219)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:507)
        at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1080)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2278)
        at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2274)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1924)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2272)
      

      The web page at the time the test failed:

      13:31:54 E   memz: "{
          "__common__": {
              "process-name": "impalad",
              "navbar": [
                  {
                      "link": "/",
                      "title": "/"
                  },
                  {
                      "link": "/backends",
                      "title": "/backends"
                  },
                  {
                      "link": "/catalog",
                      "title": "/catalog"
                  },
                  {
                      "link": "/hadoop-varz",
                      "title": "/hadoop-varz"
                  },
                  {
                      "link": "/log_level",
                      "title": "/log_level"
                  },
                  {
                      "link": "/logs",
                      "title": "/logs"
                  },
                  {
                      "link": "/memz",
                      "title": "/memz"
                  },
                  {
                      "link": "/metrics",
                      "title": "/metrics"
                  },
                  {
                      "link": "/queries",
                      "title": "/queries"
                  },
                  {
                      "link": "/rpcz",
                      "title": "/rpcz"
                  },
                  {
                      "link": "/sessions",
                      "title": "/sessions"
                  },
                  {
                      "link": "/threadz",
                      "title": "/threadz"
                  },
                  {
                      "link": "/varz",
                      "title": "/varz"
                  }
              ]
          },
          "mem_limit": "12.00 GB",
          "consumption": "2.14 GB",
          "overview": "------------------------------------------------\nMALLOC:      815880000 (  778.1 MiB) Bytes in use by application\nMALLOC: +            0 (    0.0 MiB) Bytes in page heap freelist\nMALLOC: +     31888784 (   30.4 MiB) Bytes in central cache freelist\nMALLOC: +     47000736 (   44.8 MiB) Bytes in transfer cache freelist\nMALLOC: +     93660816 (   89.3 MiB) Bytes in thread cache freelists\nMALLOC: +     16904384 (   16.1 MiB) Bytes in malloc metadata\nMALLOC:   ------------\nMALLOC: =   1005334720 (  958.8 MiB) Actual memory used (physical + swap)\nMALLOC: +   7214391296 ( 6880.2 MiB) Bytes released to OS (aka unmapped)\nMALLOC:   ------------\nMALLOC: =   8219726016 ( 7838.9 MiB) Virtual address space used\nMALLOC:\nMALLOC:          20350              Spans in use\nMALLOC:            218              Thread heaps in use\nMALLOC:           8192              Tcmalloc page size\n------------------------------------------------\nCall ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).\nBytes released to the OS take up virtual address space but no physical memory.\n",
          "detailed": "Process: Limit=12.00 GB Total=942.64 MB Peak=7.57 GB\n  Buffer Pool: Free Buffers: Total=72.00 KB\n  Buffer Pool: Clean Pages: Total=0\n  Buffer Pool: Unused Reservation: Total=-160.00 KB\n  Data Stream Service Queue: Limit=614.40 MB Total=0 Peak=77.29 KB\n  Data Stream Manager Early RPCs: Total=0 Peak=7.14 KB\n  TCMalloc Overhead: Total=164.55 MB\n  RequestPool=fe-eval-exprs: Total=0 Peak=52.83 KB\n  RequestPool=default-pool: Total=17.36 MB Peak=4.72 GB\n    Query(1946493a28ec3cab:84bd1cdf00000000): Limit=64.00 MB Reservation=0 ReservationLimit=32.00 MB OtherMemory=17.36 MB Total=17.36 MB Peak=1.96 GB\n      Fragment 1946493a28ec3cab:84bd1cdf00000008: Reservation=0 OtherMemory=17.36 MB Total=17.36 MB Peak=1.96 GB\n        EXCHANGE_NODE (id=4): Reservation=17.34 MB OtherMemory=0 Total=17.34 MB Peak=17.34 MB\n          KrpcDeferredRpcs: Total=0 Peak=0\n        HdfsTableSink: Total=0 Peak=1.94 GB\n        CodeGen: Total=0 Peak=0\n      Fragment 1946493a28ec3cab:84bd1cdf00000005: Reservation=0 OtherMemory=0 Total=0 Peak=2.93 MB\n        NESTED_LOOP_JOIN_NODE (id=2): Total=0 Peak=490.00 KB\n          Nested Loop Join Builder: Total=0 Peak=216.00 KB\n        HDFS_SCAN_NODE (id=0): Reservation=0 OtherMemory=0 Total=0 Peak=1.95 MB\n        EXCHANGE_NODE (id=3): Reservation=0 OtherMemory=0 Total=0 Peak=384.00 KB\n          KrpcDeferredRpcs: Total=0 Peak=0\n        KrpcDataStreamSender (dst_id=4): Total=0 Peak=92.44 KB\n        CodeGen: Total=0 Peak=834.50 KB\n      Fragment 1946493a28ec3cab:84bd1cdf00000002: Reservation=0 OtherMemory=0 Total=0 Peak=439.72 KB\n        HDFS_SCAN_NODE (id=1): Reservation=0 OtherMemory=0 Total=0 Peak=352.00 KB\n        KrpcDataStreamSender (dst_id=3): Total=0 Peak=76.78 KB\n        CodeGen: Total=0 Peak=210.50 KB\n  Untracked Memory: Total=760.81 MB",
          "systeminfo": "Physical Memory: 66.23 GB\nTransparent Huge Pages Config:\n  enabled: <unknown>\n  defrag: <unknown>\n  khugepaged defrag: <unknown>\n",
          "aggregate_metrics": [
              {
                  "name": "memory.anon-huge-page-bytes",
                  "description": "Total bytes of anonymous (a.k.a. transparent) huge pages used by this process.",
                  "human_readable": "0",
                  "value": 0,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "memory.mapped-bytes",
                  "description": "Total bytes of memory mappings in this process (the virtual memory size).",
                  "human_readable": "28.26 GB",
                  "value": 30344564736,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "memory.num-maps",
                  "description": "Total number of memory mappings in this process.",
                  "human_readable": "989",
                  "value": 989,
                  "kind": "GAUGE",
                  "units": "NONE"
              },
              {
                  "name": "memory.rss",
                  "description": "Resident set size (RSS) of this process, including TCMalloc, buffer pool and Jvm.",
                  "human_readable": "1.46 GB",
                  "value": 1567739904,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "memory.thp.defrag",
                  "description": "The system-wide 'defrag' setting for Transparent Huge Pages.",
                  "human_readable": "<unknown>",
                  "value": "<unknown>",
                  "kind": "PROPERTY",
                  "units": "NONE"
              },
              {
                  "name": "memory.thp.enabled",
                  "description": "The system-wide 'enabled' setting for Transparent Huge Pages.",
                  "human_readable": "<unknown>",
                  "value": "<unknown>",
                  "kind": "PROPERTY",
                  "units": "NONE"
              },
              {
                  "name": "memory.thp.khugepaged-defrag",
                  "description": "The system-wide 'defrag' setting for khugepaged.",
                  "human_readable": "<unknown>",
                  "value": "<unknown>",
                  "kind": "PROPERTY",
                  "units": "NONE"
              },
              {
                  "name": "memory.total-used",
                  "description": "Total memory currently used by TCMalloc and buffer pool.",
                  "human_readable": "942.64 MB",
                  "value": 980697088,
                  "kind": "GAUGE",
                  "units": "BYTES"
              }
          ],
          "jvm_total": [
              {
                  "name": "jvm.total.committed-usage-bytes",
                  "description": "Jvm total Committed Usage Bytes",
                  "human_readable": "203.75 MB",
                  "value": 213647360,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.total.current-usage-bytes",
                  "description": "Jvm total Current Usage Bytes",
                  "human_readable": "189.76 MB",
                  "value": 199001896,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.total.init-usage-bytes",
                  "description": "Jvm total Init Usage Bytes",
                  "human_readable": "1.04 GB",
                  "value": 1113522176,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.total.max-usage-bytes",
                  "description": "Jvm total Max Usage Bytes",
                  "human_readable": "17.78 GB",
                  "value": 19091947519,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.total.peak-committed-usage-bytes",
                  "description": "Jvm total Peak Committed Usage Bytes",
                  "human_readable": "1.23 GB",
                  "value": 1315700736,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.total.peak-current-usage-bytes",
                  "description": "Jvm total Peak Current Usage Bytes",
                  "human_readable": "231.21 MB",
                  "value": 242445032,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.total.peak-init-usage-bytes",
                  "description": "Jvm total Peak Init Usage Bytes",
                  "human_readable": "1.04 GB",
                  "value": 1113522176,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.total.peak-max-usage-bytes",
                  "description": "Jvm total Peak Max Usage Bytes",
                  "human_readable": "17.80 GB",
                  "value": 19113967615,
                  "kind": "GAUGE",
                  "units": "BYTES"
              }
          ],
          "jvm_heap": [
              {
                  "name": "jvm.heap.committed-usage-bytes",
                  "description": "Jvm heap Committed Usage Bytes",
                  "human_readable": "97.50 MB",
                  "value": 102236160,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.heap.current-usage-bytes",
                  "description": "Jvm heap Current Usage Bytes",
                  "human_readable": "84.29 MB",
                  "value": 88385256,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.heap.init-usage-bytes",
                  "description": "Jvm heap Init Usage Bytes",
                  "human_readable": "1.04 GB",
                  "value": 1111490560,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.heap.max-usage-bytes",
                  "description": "Jvm heap Max Usage Bytes",
                  "human_readable": "14.72 GB",
                  "value": 15802564608,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.heap.peak-committed-usage-bytes",
                  "description": "Jvm heap Peak Committed Usage Bytes",
                  "human_readable": "0",
                  "value": 0,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.heap.peak-current-usage-bytes",
                  "description": "Jvm heap Peak Current Usage Bytes",
                  "human_readable": "0",
                  "value": 0,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.heap.peak-init-usage-bytes",
                  "description": "Jvm heap Peak Init Usage Bytes",
                  "human_readable": "0",
                  "value": 0,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.heap.peak-max-usage-bytes",
                  "description": "Jvm heap Peak Max Usage Bytes",
                  "human_readable": "0",
                  "value": 0,
                  "kind": "GAUGE",
                  "units": "BYTES"
              }
          ],
          "jvm_non_heap": [
              {
                  "name": "jvm.non-heap.committed-usage-bytes",
                  "description": "Jvm non-heap Committed Usage Bytes",
                  "human_readable": "106.25 MB",
                  "value": 111411200,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.non-heap.current-usage-bytes",
                  "description": "Jvm non-heap Current Usage Bytes",
                  "human_readable": "105.31 MB",
                  "value": 110427856,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.non-heap.init-usage-bytes",
                  "description": "Jvm non-heap Init Usage Bytes",
                  "human_readable": "2.44 MB",
                  "value": 2555904,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.non-heap.max-usage-bytes",
                  "description": "Jvm non-heap Max Usage Bytes",
                  "human_readable": "-1.00 B",
                  "value": -1,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.non-heap.peak-committed-usage-bytes",
                  "description": "Jvm non-heap Peak Committed Usage Bytes",
                  "human_readable": "0",
                  "value": 0,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.non-heap.peak-current-usage-bytes",
                  "description": "Jvm non-heap Peak Current Usage Bytes",
                  "human_readable": "0",
                  "value": 0,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.non-heap.peak-init-usage-bytes",
                  "description": "Jvm non-heap Peak Init Usage Bytes",
                  "human_readable": "0",
                  "value": 0,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "jvm.non-heap.peak-max-usage-bytes",
                  "description": "Jvm non-heap Peak Max Usage Bytes",
                  "human_readable": "0",
                  "value": 0,
                  "kind": "GAUGE",
                  "units": "BYTES"
              }
          ],
          "buffer_pool": [
              {
                  "name": "buffer-pool.clean-page-bytes",
                  "description": "Total bytes of clean page memory cached in the buffer pool.",
                  "human_readable": "0",
                  "value": 0,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "buffer-pool.clean-pages",
                  "description": "Total number of clean pages cached in the buffer pool.",
                  "human_readable": "0",
                  "value": 0,
                  "kind": "GAUGE",
                  "units": "NONE"
              },
              {
                  "name": "buffer-pool.clean-pages-limit",
                  "description": "Limit on number of clean pages cached in the buffer pool.",
                  "human_readable": "1.02 GB",
                  "value": 1095216332,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "buffer-pool.free-buffer-bytes",
                  "description": "Total bytes of free buffer memory cached in the buffer pool.",
                  "human_readable": "72.00 KB",
                  "value": 73728,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "buffer-pool.free-buffers",
                  "description": "Total number of free buffers cached in the buffer pool.",
                  "human_readable": "3",
                  "value": 3,
                  "kind": "GAUGE",
                  "units": "NONE"
              },
              {
                  "name": "buffer-pool.limit",
                  "description": "Maximum allowed bytes allocated by the buffer pool.",
                  "human_readable": "10.20 GB",
                  "value": 10952163328,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "buffer-pool.reserved",
                  "description": "Total bytes of buffers reserved by Impala subsystems",
                  "human_readable": "17.34 MB",
                  "value": 18186240,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "buffer-pool.system-allocated",
                  "description": "Total buffer memory currently allocated by the buffer pool.",
                  "human_readable": "17.26 MB",
                  "value": 18096128,
                  "kind": "GAUGE",
                  "units": "BYTES"
              },
              {
                  "name": "buffer-pool.unused-reservation-bytes",
                  "description": "Total bytes of buffer reservations by Impala subsystems that are currently unused",
                  "human_readable": "160.00 KB",
                  "value": 163840,
                  "kind": "GAUGE",
                  "units": "BYTES"
              }
          ]
      }"
      13:31:54 E   metrics: "{
          "__common__": {
              "process-name": "impalad",
              "navbar": [
                  {
                      "link": "/",
                      "title": "/"
                  },
                  {
                      "link": "/backends",
                      "title": "/backends"
                  },
                  {
                      "link": "/catalog",
                      "title": "/catalog"
                  },
                  {
                      "link": "/hadoop-varz",
                      "title": "/hadoop-varz"
                  },
                  {
                      "link": "/log_level",
                      "title": "/log_level"
                  },
                  {
                      "link": "/logs",
                      "title": "/logs"
                  },
                  {
                      "link": "/memz",
                      "title": "/memz"
                  },
                  {
                      "link": "/metrics",
                      "title": "/metrics"
                  },
                  {
                      "link": "/queries",
                      "title": "/queries"
                  },
                  {
                      "link": "/rpcz",
                      "title": "/rpcz"
                  },
                  {
                      "link": "/sessions",
                      "title": "/sessions"
                  },
                  {
                      "link": "/threadz",
                      "title": "/threadz"
                  },
                  {
                      "link": "/varz",
                      "title": "/varz"
                  }
              ]
          },
          "metric_group": {
              "metrics": [
                  {
                      "name": "catalog.server.client-cache.clients-in-use",
                      "description": "The number of clients currently in use by the Catalog Server client cache.",
                      "human_readable": "0",
                      "value": 0,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "catalog.server.client-cache.total-clients",
                      "description": "The total number of clients in the Catalog Server client cache.",
                      "human_readable": "2",
                      "value": 2,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "external-data-source.class-cache.hits",
                      "description": "Number of cache hits in the External Data Source Class Cache",
                      "human_readable": "3",
                      "value": 3,
                      "kind": "COUNTER",
                      "units": "UNIT"
                  },
                  {
                      "name": "external-data-source.class-cache.misses",
                      "description": "Number of cache misses in the External Data Source Class Cache",
                      "human_readable": "5",
                      "value": 5,
                      "kind": "COUNTER",
                      "units": "UNIT"
                  },
                  {
                      "name": "impala-server.backends.client-cache.clients-in-use",
                      "description": "The number of active Impala Backend clients. These clients are for communication with other Impala Daemons.",
                      "human_readable": "0",
                      "value": 0,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "impala-server.backends.client-cache.total-clients",
                      "description": "The total number of Impala Backend clients in this Impala Daemon's client cache. These clients are for communication with other Impala Daemons.",
                      "human_readable": "7",
                      "value": 7,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "impala.thrift-server.backend.connection-setup-queue-size",
                      "description": "The number of connections to the Impala Backend Server that have been accepted and are waiting to be setup.",
                      "human_readable": "0",
                      "value": 0,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "impala.thrift-server.backend.connections-in-use",
                      "description": "The number of active Impala Backend client connections to this Impala Daemon.",
                      "human_readable": "8",
                      "value": 8,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "impala.thrift-server.backend.total-connections",
                      "description": "The total number of Impala Backend client connections made to this Impala Daemon over its lifetime.",
                      "human_readable": "8",
                      "value": 8,
                      "kind": "COUNTER",
                      "units": "UNIT"
                  },
                  {
                      "name": "impala.thrift-server.beeswax-frontend.connection-setup-queue-size",
                      "description": "The number of Beeswax API connections to this Impala Daemon that have been accepted and are waiting to be setup.",
                      "human_readable": "0",
                      "value": 0,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "impala.thrift-server.beeswax-frontend.connections-in-use",
                      "description": "The number of active Beeswax API connections to this Impala Daemon.",
                      "human_readable": "1",
                      "value": 1,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "impala.thrift-server.beeswax-frontend.total-connections",
                      "description": "The total number of Beeswax API connections made to this Impala Daemon over its lifetime.",
                      "human_readable": "31",
                      "value": 31,
                      "kind": "COUNTER",
                      "units": "UNIT"
                  },
                  {
                      "name": "impala.thrift-server.hiveserver2-frontend.connection-setup-queue-size",
                      "description": "The number of HiveServer2 API connections to this Impala Daemon that have been accepted and are waiting to be setup.",
                      "human_readable": "0",
                      "value": 0,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "impala.thrift-server.hiveserver2-frontend.connections-in-use",
                      "description": "The number of active HiveServer2 API connections to this Impala Daemon.",
                      "human_readable": "0",
                      "value": 0,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "impala.thrift-server.hiveserver2-frontend.total-connections",
                      "description": "The total number of HiveServer2 API connections made to this Impala Daemon over its lifetime.",
                      "human_readable": "15",
                      "value": 15,
                      "kind": "COUNTER",
                      "units": "UNIT"
                  },
                  {
                      "name": "kudu-client.version",
                      "description": "A version string identifying the Kudu client",
                      "human_readable": "kudu 1.8.0-SNAPSHOT (rev a95441846bb1381ff2543ffcddf3854184a111ea)",
                      "value": "kudu 1.8.0-SNAPSHOT (rev a95441846bb1381ff2543ffcddf3854184a111ea)",
                      "kind": "PROPERTY",
                      "units": "NONE"
                  },
                  {
                      "name": "mem-tracker.process.bytes-freed-by-last-gc",
                      "description": "The amount of memory freed by the last memory tracker garbage collection.",
                      "human_readable": "-1.00 B",
                      "value": -1,
                      "kind": "GAUGE",
                      "units": "BYTES"
                  },
                  {
                      "name": "mem-tracker.process.bytes-over-limit",
                      "description": "The amount of memory by which the process was over its memory limit the last time the memory limit was encountered.",
                      "human_readable": "-1.00 B",
                      "value": -1,
                      "kind": "GAUGE",
                      "units": "BYTES"
                  },
                  {
                      "name": "mem-tracker.process.limit",
                      "description": "The process memory tracker limit.",
                      "human_readable": "12.00 GB",
                      "value": 12884901888,
                      "kind": "GAUGE",
                      "units": "BYTES"
                  },
                  {
                      "name": "mem-tracker.process.num-gcs",
                      "description": "The total number of garbage collections performed by the memory tracker over the life of the process.",
                      "human_readable": "0",
                      "value": 0,
                      "kind": "COUNTER",
                      "units": "UNIT"
                  },
                  {
                      "name": "process-start-time",
                      "description": "The local start time of the process",
                      "human_readable": "2018-06-13 09:40:46.500186000",
                      "value": "2018-06-13 09:40:46.500186000",
                      "kind": "PROPERTY",
                      "units": "NONE"
                  },
                  {
                      "name": "request-pool-service.resolve-pool-duration-ms",
                      "description": "Time (ms) spent resolving request request pools.",
                      "human_readable": "count: 0",
                      "units": "TIME_MS",
                      "count": 0
                  },
                  {
                      "name": "rpc-method.backend.ImpalaInternalService.CancelQueryFInstances.call_duration",
                      "description": "",
                      "human_readable": "Count: 291, min / max: 0 / 46ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 5ms, 99.9th %-ile: 46ms",
                      "25th %-ile": 0,
                      "50th %-ile": 0,
                      "75th %-ile": 1,
                      "90th %-ile": 1,
                      "95th %-ile": 5,
                      "99.9th %-ile": 46,
                      "max": 46,
                      "min": 0,
                      "count": 291,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.backend.ImpalaInternalService.ExecQueryFInstances.call_duration",
                      "description": "",
                      "human_readable": "Count: 1866, min / max: 0 / 25ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 2ms, 95th %-ile: 2ms, 99.9th %-ile: 21ms",
                      "25th %-ile": 1,
                      "50th %-ile": 1,
                      "75th %-ile": 1,
                      "90th %-ile": 2,
                      "95th %-ile": 2,
                      "99.9th %-ile": 21,
                      "max": 25,
                      "min": 0,
                      "count": 1866,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.backend.ImpalaInternalService.PublishFilter.call_duration",
                      "description": "",
                      "human_readable": "Count: 13, min / max: 4ms / 6ms, 25th %-ile: 5ms, 50th %-ile: 6ms, 75th %-ile: 6ms, 90th %-ile: 6ms, 95th %-ile: 6ms, 99.9th %-ile: 6ms",
                      "25th %-ile": 5,
                      "50th %-ile": 6,
                      "75th %-ile": 6,
                      "90th %-ile": 6,
                      "95th %-ile": 6,
                      "99.9th %-ile": 6,
                      "max": 6,
                      "min": 4,
                      "count": 13,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.backend.ImpalaInternalService.ReportExecStatus.call_duration",
                      "description": "",
                      "human_readable": "Count: 8046, min / max: 0 / 51ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 2ms, 99.9th %-ile: 37ms",
                      "25th %-ile": 1,
                      "50th %-ile": 1,
                      "75th %-ile": 1,
                      "90th %-ile": 1,
                      "95th %-ile": 2,
                      "99.9th %-ile": 37,
                      "max": 51,
                      "min": 0,
                      "count": 8046,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.backend.ImpalaInternalService.UpdateFilter.call_duration",
                      "description": "",
                      "human_readable": "Count: 27, min / max: 3ms / 24ms, 25th %-ile: 4ms, 50th %-ile: 4ms, 75th %-ile: 21ms, 90th %-ile: 23ms, 95th %-ile: 23ms, 99.9th %-ile: 24ms",
                      "25th %-ile": 4,
                      "50th %-ile": 4,
                      "75th %-ile": 21,
                      "90th %-ile": 23,
                      "95th %-ile": 23,
                      "99.9th %-ile": 24,
                      "max": 24,
                      "min": 3,
                      "count": 27,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.beeswax.BeeswaxService.close.call_duration",
                      "description": "",
                      "human_readable": "Count: 3653, min / max: 0 / 85ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 13ms, 90th %-ile: 21ms, 95th %-ile: 27ms, 99.9th %-ile: 68ms",
                      "25th %-ile": 1,
                      "50th %-ile": 1,
                      "75th %-ile": 13,
                      "90th %-ile": 21,
                      "95th %-ile": 27,
                      "99.9th %-ile": 68,
                      "max": 85,
                      "min": 0,
                      "count": 3653,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.beeswax.BeeswaxService.fetch.call_duration",
                      "description": "",
                      "human_readable": "Count: 3086, min / max: 0 / 25s223ms, 25th %-ile: 0, 50th %-ile: 1ms, 75th %-ile: 6ms, 90th %-ile: 24ms, 95th %-ile: 37ms, 99.9th %-ile: 462ms",
                      "25th %-ile": 0,
                      "50th %-ile": 1,
                      "75th %-ile": 6,
                      "90th %-ile": 24,
                      "95th %-ile": 37,
                      "99.9th %-ile": 462,
                      "max": 25223,
                      "min": 0,
                      "count": 3086,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.beeswax.BeeswaxService.get_default_configuration.call_duration",
                      "description": "",
                      "human_readable": "Count: 1, min / max: 2ms / 2ms, 25th %-ile: 2ms, 50th %-ile: 2ms, 75th %-ile: 2ms, 90th %-ile: 2ms, 95th %-ile: 2ms, 99.9th %-ile: 2ms",
                      "25th %-ile": 2,
                      "50th %-ile": 2,
                      "75th %-ile": 2,
                      "90th %-ile": 2,
                      "95th %-ile": 2,
                      "99.9th %-ile": 2,
                      "max": 2,
                      "min": 2,
                      "count": 1,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.beeswax.BeeswaxService.get_log.call_duration",
                      "description": "",
                      "human_readable": "Count: 3891, min / max: 0 / 2ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                      "25th %-ile": 0,
                      "50th %-ile": 0,
                      "75th %-ile": 0,
                      "90th %-ile": 1,
                      "95th %-ile": 1,
                      "99.9th %-ile": 1,
                      "max": 2,
                      "min": 0,
                      "count": 3891,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.beeswax.BeeswaxService.get_results_metadata.call_duration",
                      "description": "",
                      "human_readable": "Count: 2209, min / max: 0 / 3ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                      "25th %-ile": 0,
                      "50th %-ile": 0,
                      "75th %-ile": 0,
                      "90th %-ile": 1,
                      "95th %-ile": 1,
                      "99.9th %-ile": 1,
                      "max": 3,
                      "min": 0,
                      "count": 2209,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.beeswax.BeeswaxService.get_state.call_duration",
                      "description": "",
                      "human_readable": "Count: 23316, min / max: 0 / 2ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                      "25th %-ile": 0,
                      "50th %-ile": 0,
                      "75th %-ile": 0,
                      "90th %-ile": 0,
                      "95th %-ile": 1,
                      "99.9th %-ile": 1,
                      "max": 2,
                      "min": 0,
                      "count": 23316,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.beeswax.BeeswaxService.query.call_duration",
                      "description": "",
                      "human_readable": "Count: 4097, min / max: 5ms / 29s267ms, 25th %-ile: 9ms, 50th %-ile: 18ms, 75th %-ile: 34ms, 90th %-ile: 3s768ms, 95th %-ile: 5s324ms, 99.9th %-ile: 10s248ms",
                      "25th %-ile": 9,
                      "50th %-ile": 18,
                      "75th %-ile": 34,
                      "90th %-ile": 3768,
                      "95th %-ile": 5324,
                      "99.9th %-ile": 10248,
                      "max": 29267,
                      "min": 5,
                      "count": 4097,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.beeswax.ImpalaService.Cancel.call_duration",
                      "description": "",
                      "human_readable": "Count: 74, min / max: 1ms / 6ms, 25th %-ile: 2ms, 50th %-ile: 3ms, 75th %-ile: 4ms, 90th %-ile: 5ms, 95th %-ile: 5ms, 99.9th %-ile: 6ms",
                      "25th %-ile": 2,
                      "50th %-ile": 3,
                      "75th %-ile": 4,
                      "90th %-ile": 5,
                      "95th %-ile": 5,
                      "99.9th %-ile": 6,
                      "max": 6,
                      "min": 1,
                      "count": 74,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.beeswax.ImpalaService.CloseInsert.call_duration",
                      "description": "",
                      "human_readable": "Count: 316, min / max: 3ms / 63ms, 25th %-ile: 5ms, 50th %-ile: 10ms, 75th %-ile: 16ms, 90th %-ile: 20ms, 95th %-ile: 23ms, 99.9th %-ile: 63ms",
                      "25th %-ile": 5,
                      "50th %-ile": 10,
                      "75th %-ile": 16,
                      "90th %-ile": 20,
                      "95th %-ile": 23,
                      "99.9th %-ile": 63,
                      "max": 63,
                      "min": 3,
                      "count": 316,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.beeswax.ImpalaService.GetExecSummary.call_duration",
                      "description": "",
                      "human_readable": "Count: 3704, min / max: 0 / 2ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 2ms",
                      "25th %-ile": 0,
                      "50th %-ile": 0,
                      "75th %-ile": 0,
                      "90th %-ile": 1,
                      "95th %-ile": 1,
                      "99.9th %-ile": 2,
                      "max": 2,
                      "min": 0,
                      "count": 3704,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.beeswax.ImpalaService.GetRuntimeProfile.call_duration",
                      "description": "",
                      "human_readable": "Count: 3708, min / max: 0 / 1s899ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 3ms, 90th %-ile: 5ms, 95th %-ile: 7ms, 99.9th %-ile: 14ms",
                      "25th %-ile": 0,
                      "50th %-ile": 0,
                      "75th %-ile": 3,
                      "90th %-ile": 5,
                      "95th %-ile": 7,
                      "99.9th %-ile": 14,
                      "max": 1899,
                      "min": 0,
                      "count": 3708,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.CloseOperation.call_duration",
                      "description": "",
                      "human_readable": "Count: 50, min / max: 0 / 14ms, 25th %-ile: 1ms, 50th %-ile: 2ms, 75th %-ile: 4ms, 90th %-ile: 11ms, 95th %-ile: 12ms, 99.9th %-ile: 14ms",
                      "25th %-ile": 1,
                      "50th %-ile": 2,
                      "75th %-ile": 4,
                      "90th %-ile": 11,
                      "95th %-ile": 12,
                      "99.9th %-ile": 14,
                      "max": 14,
                      "min": 0,
                      "count": 50,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.CloseSession.call_duration",
                      "description": "",
                      "human_readable": "Count: 9, min / max: 0 / 5ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 1ms, 90th %-ile: 3ms, 95th %-ile: 5ms, 99.9th %-ile: 5ms",
                      "25th %-ile": 0,
                      "50th %-ile": 0,
                      "75th %-ile": 1,
                      "90th %-ile": 3,
                      "95th %-ile": 5,
                      "99.9th %-ile": 5,
                      "max": 5,
                      "min": 0,
                      "count": 9,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.ExecuteStatement.call_duration",
                      "description": "",
                      "human_readable": "Count: 48, min / max: 0 / 666ms, 25th %-ile: 12ms, 50th %-ile: 14ms, 75th %-ile: 21ms, 90th %-ile: 69ms, 95th %-ile: 288ms, 99.9th %-ile: 666ms",
                      "25th %-ile": 12,
                      "50th %-ile": 14,
                      "75th %-ile": 21,
                      "90th %-ile": 69,
                      "95th %-ile": 288,
                      "99.9th %-ile": 666,
                      "max": 666,
                      "min": 0,
                      "count": 48,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.FetchResults.call_duration",
                      "description": "",
                      "human_readable": "Count: 114, min / max: 0 / 34ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 1ms, 90th %-ile: 4ms, 95th %-ile: 6ms, 99.9th %-ile: 34ms",
                      "25th %-ile": 0,
                      "50th %-ile": 0,
                      "75th %-ile": 1,
                      "90th %-ile": 4,
                      "95th %-ile": 6,
                      "99.9th %-ile": 34,
                      "max": 34,
                      "min": 0,
                      "count": 114,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.GetCatalogs.call_duration",
                      "description": "",
                      "human_readable": "Count: 1, min / max: 19ms / 19ms, 25th %-ile: 19ms, 50th %-ile: 19ms, 75th %-ile: 19ms, 90th %-ile: 19ms, 95th %-ile: 19ms, 99.9th %-ile: 19ms",
                      "25th %-ile": 19,
                      "50th %-ile": 19,
                      "75th %-ile": 19,
                      "90th %-ile": 19,
                      "95th %-ile": 19,
                      "99.9th %-ile": 19,
                      "max": 19,
                      "min": 19,
                      "count": 1,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.GetColumns.call_duration",
                      "description": "",
                      "human_readable": "Count: 15, min / max: 5ms / 4s001ms, 25th %-ile: 5ms, 50th %-ile: 6ms, 75th %-ile: 3s406ms, 90th %-ile: 3s924ms, 95th %-ile: 3s924ms, 99.9th %-ile: 4s000ms",
                      "25th %-ile": 5,
                      "50th %-ile": 6,
                      "75th %-ile": 3406,
                      "90th %-ile": 3924,
                      "95th %-ile": 3924,
                      "99.9th %-ile": 4000,
                      "max": 4001,
                      "min": 5,
                      "count": 15,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.GetFunctions.call_duration",
                      "description": "",
                      "human_readable": "Count: 2, min / max: 5ms / 22ms, 25th %-ile: 5ms, 50th %-ile: 5ms, 75th %-ile: 22ms, 90th %-ile: 22ms, 95th %-ile: 22ms, 99.9th %-ile: 22ms",
                      "25th %-ile": 5,
                      "50th %-ile": 5,
                      "75th %-ile": 22,
                      "90th %-ile": 22,
                      "95th %-ile": 22,
                      "99.9th %-ile": 22,
                      "max": 22,
                      "min": 5,
                      "count": 2,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.GetOperationStatus.call_duration",
                      "description": "",
                      "human_readable": "Count: 607, min / max: 0 / 1ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                      "25th %-ile": 0,
                      "50th %-ile": 0,
                      "75th %-ile": 0,
                      "90th %-ile": 0,
                      "95th %-ile": 1,
                      "99.9th %-ile": 1,
                      "max": 1,
                      "min": 0,
                      "count": 607,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.GetResultSetMetadata.call_duration",
                      "description": "",
                      "human_readable": "Count: 49, min / max: 0 / 1ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                      "25th %-ile": 0,
                      "50th %-ile": 0,
                      "75th %-ile": 1,
                      "90th %-ile": 1,
                      "95th %-ile": 1,
                      "99.9th %-ile": 1,
                      "max": 1,
                      "min": 0,
                      "count": 49,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.GetSchemas.call_duration",
                      "description": "",
                      "human_readable": "Count: 1, min / max: 75ms / 75ms, 25th %-ile: 75ms, 50th %-ile: 75ms, 75th %-ile: 75ms, 90th %-ile: 75ms, 95th %-ile: 75ms, 99.9th %-ile: 75ms",
                      "25th %-ile": 75,
                      "50th %-ile": 75,
                      "75th %-ile": 75,
                      "90th %-ile": 75,
                      "95th %-ile": 75,
                      "99.9th %-ile": 75,
                      "max": 75,
                      "min": 75,
                      "count": 1,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.GetTableTypes.call_duration",
                      "description": "",
                      "human_readable": "Count: 1, min / max: 62ms / 62ms, 25th %-ile: 62ms, 50th %-ile: 62ms, 75th %-ile: 62ms, 90th %-ile: 62ms, 95th %-ile: 62ms, 99.9th %-ile: 62ms",
                      "25th %-ile": 62,
                      "50th %-ile": 62,
                      "75th %-ile": 62,
                      "90th %-ile": 62,
                      "95th %-ile": 62,
                      "99.9th %-ile": 62,
                      "max": 62,
                      "min": 62,
                      "count": 1,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.GetTables.call_duration",
                      "description": "",
                      "human_readable": "Count: 10, min / max: 4ms / 22ms, 25th %-ile: 5ms, 50th %-ile: 7ms, 75th %-ile: 18ms, 90th %-ile: 18ms, 95th %-ile: 22ms, 99.9th %-ile: 22ms",
                      "25th %-ile": 5,
                      "50th %-ile": 7,
                      "75th %-ile": 18,
                      "90th %-ile": 18,
                      "95th %-ile": 22,
                      "99.9th %-ile": 22,
                      "max": 22,
                      "min": 4,
                      "count": 10,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "rpc-method.hs2.TCLIService.OpenSession.call_duration",
                      "description": "",
                      "human_readable": "Count: 17, min / max: 0 / 1ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                      "25th %-ile": 0,
                      "50th %-ile": 0,
                      "75th %-ile": 1,
                      "90th %-ile": 1,
                      "95th %-ile": 1,
                      "99.9th %-ile": 1,
                      "max": 1,
                      "min": 0,
                      "count": 17,
                      "kind": "HISTOGRAM",
                      "units": "TIME_MS"
                  },
                  {
                      "name": "statestore-subscriber.registration-id",
                      "description": "The most recent registration ID for this subscriber with the statestore. Set to 'N/A' if no registration has been completed",
                      "human_readable": "7746be91b882d215:e84fb5b1335ead91",
                      "value": "7746be91b882d215:e84fb5b1335ead91",
                      "kind": "PROPERTY",
                      "units": "NONE"
                  },
                  {
                      "name": "statestore-subscriber.statestore.client-cache.clients-in-use",
                      "description": "The number of active StateStore subscriber clients in this Impala Daemon's client cache. These clients are for communication from this role to the StateStore.",
                      "human_readable": "0",
                      "value": 0,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "statestore-subscriber.statestore.client-cache.total-clients",
                      "description": "The total number of StateStore subscriber clients in this Impala Daemon's client cache. These clients are for communication from this role to the StateStore.",
                      "human_readable": "1",
                      "value": 1,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "thread-manager.running-threads",
                      "description": "The number of running threads in this process.",
                      "human_readable": "30",
                      "value": 30,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "thread-manager.total-threads-created",
                      "description": "Threads created over the lifetime of the process.",
                      "human_readable": "17025",
                      "value": 17025,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "tmp-file-mgr.active-scratch-dirs",
                      "description": "The number of active scratch directories for spilling to disk.",
                      "human_readable": "1",
                      "value": 1,
                      "kind": "GAUGE",
                      "units": "NONE"
                  },
                  {
                      "name": "tmp-file-mgr.active-scratch-dirs.list",
                      "description": "The set of all active scratch directories for spilling to disk.",
                      "human_readable": "[/tmp/impala-scratch]",
                      "items": [
                          "/tmp/impala-scratch"
                      ]
                  }
              ],
              "name": "impala-metrics",
              "child_groups": [
                  {
                      "metrics": [
                          {
                              "name": "admission-controller.agg-mem-reserved.default-pool",
                              "description": "Resource Pool default-pool Aggregate Mem Reserved",
                              "human_readable": "201326592",
                              "value": 201326592,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "admission-controller.agg-num-queued.default-pool",
                              "description": "Resource Pool default-pool Aggregate Queue Size",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "admission-controller.agg-num-running.default-pool",
                              "description": "Resource Pool default-pool Aggregate Num Running",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "admission-controller.local-backend-mem-reserved.default-pool",
                              "description": "Resource Pool default-pool Mem Reserved by the backend coordinator",
                              "human_readable": "67108864",
                              "value": 67108864,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "admission-controller.local-backend-mem-usage.default-pool",
                              "description": "Resource Pool default-pool Coordinator Backend Mem Usage",
                              "human_readable": "18202624",
                              "value": 18202624,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "admission-controller.local-mem-admitted.default-pool",
                              "description": "Resource Pool default-pool Local Mem Admitted",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "admission-controller.local-num-admitted-running.default-pool",
                              "description": "Resource Pool default-pool Coordinator Backend Num Running",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "admission-controller.local-num-queued.default-pool",
                              "description": "Resource Pool default-pool Queue Size on the coordinator",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "admission-controller.pool-max-mem-resources.default-pool",
                              "description": "Resource Pool default-pool Configured Max Mem Resources",
                              "human_readable": "-1",
                              "value": -1,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "admission-controller.pool-max-queued.default-pool",
                              "description": "Resource Pool default-pool Configured Max Queued",
                              "human_readable": "200",
                              "value": 200,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "admission-controller.pool-max-requests.default-pool",
                              "description": "Resource Pool default-pool Configured Max Requests",
                              "human_readable": "-1",
                              "value": -1,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "admission-controller.time-in-queue-ms.default-pool",
                              "description": "Resource Pool default-pool Time in Queue",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "TIME_MS"
                          },
                          {
                              "name": "admission-controller.total-admitted.default-pool",
                              "description": "Total number of requests admitted to pool default-pool",
                              "human_readable": "1.69K",
                              "value": 1690,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "admission-controller.total-dequeued.default-pool",
                              "description": "Total number of requests dequeued in pool default-pool",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "admission-controller.total-queued.default-pool",
                              "description": "Total number of requests queued in pool default-pool",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "admission-controller.total-rejected.default-pool",
                              "description": "Total number of requests rejected in pool default-pool",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "admission-controller.total-released.default-pool",
                              "description": "Total number of requests that have completed and released resources in pool default-pool",
                              "human_readable": "1.69K",
                              "value": 1690,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "admission-controller.total-timed-out.default-pool",
                              "description": "Total number of requests timed out waiting while queued in pool default-pool",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          }
                      ],
                      "name": "admission-controller",
                      "child_groups": []
                  },
                  {
                      "metrics": [
                          {
                              "name": "buffer-pool.clean-page-bytes",
                              "description": "Total bytes of clean page memory cached in the buffer pool.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "buffer-pool.clean-pages",
                              "description": "Total number of clean pages cached in the buffer pool.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "buffer-pool.clean-pages-limit",
                              "description": "Limit on number of clean pages cached in the buffer pool.",
                              "human_readable": "1.02 GB",
                              "value": 1095216332,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "buffer-pool.free-buffer-bytes",
                              "description": "Total bytes of free buffer memory cached in the buffer pool.",
                              "human_readable": "72.00 KB",
                              "value": 73728,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "buffer-pool.free-buffers",
                              "description": "Total number of free buffers cached in the buffer pool.",
                              "human_readable": "3",
                              "value": 3,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "buffer-pool.limit",
                              "description": "Maximum allowed bytes allocated by the buffer pool.",
                              "human_readable": "10.20 GB",
                              "value": 10952163328,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "buffer-pool.reserved",
                              "description": "Total bytes of buffers reserved by Impala subsystems",
                              "human_readable": "17.34 MB",
                              "value": 18186240,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "buffer-pool.system-allocated",
                              "description": "Total buffer memory currently allocated by the buffer pool.",
                              "human_readable": "17.26 MB",
                              "value": 18096128,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "buffer-pool.unused-reservation-bytes",
                              "description": "Total bytes of buffer reservations by Impala subsystems that are currently unused",
                              "human_readable": "160.00 KB",
                              "value": 163840,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          }
                      ],
                      "name": "buffer-pool",
                      "child_groups": []
                  },
                  {
                      "metrics": [
                          {
                              "name": "senders-blocked-on-recvr-creation",
                              "description": "Number of senders waiting for receiving fragment to initialize",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "total-senders-blocked-on-recvr-creation",
                              "description": "Total number of senders that have been blocked waiting for receiving fragment to initialize.",
                              "human_readable": "66",
                              "value": 66,
                              "kind": "COUNTER",
                              "units": "NONE"
                          },
                          {
                              "name": "total-senders-timedout-waiting-for-recvr-creation",
                              "description": "Total number of senders that timed-out waiting for receiving fragment to initialize.",
                              "human_readable": "66",
                              "value": 66,
                              "kind": "COUNTER",
                              "units": "NONE"
                          }
                      ],
                      "name": "datastream-manager",
                      "child_groups": []
                  },
                  {
                      "metrics": [
                          {
                              "name": "impala-server.ddl-durations-ms",
                              "description": "Distribution of DDL operation latencies",
                              "human_readable": "Count: 2172, min / max: 3ms / 42s080ms, 25th %-ile: 5ms, 50th %-ile: 6ms, 75th %-ile: 20ms, 90th %-ile: 3s784ms, 95th %-ile: 5s984ms, 99.9th %-ile: 29s264ms",
                              "25th %-ile": 5,
                              "50th %-ile": 6,
                              "75th %-ile": 20,
                              "90th %-ile": 3784,
                              "95th %-ile": 5984,
                              "99.9th %-ile": 29264,
                              "max": 42080,
                              "min": 3,
                              "count": 2172,
                              "kind": "HISTOGRAM",
                              "units": "TIME_MS"
                          },
                          {
                              "name": "impala-server.hash-table.total-bytes",
                              "description": "The current size of all allocated hash tables.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "impala-server.hedged-read-ops",
                              "description": "The total number of hedged reads attempted over the life of the process",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "impala-server.hedged-read-ops-win",
                              "description": "The total number of times hedged reads were faster than regular read operations",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "impala-server.io-mgr.bytes-read",
                              "description": "The total number of bytes read by the IO manager.",
                              "human_readable": "4.52 GB",
                              "value": 4852715212,
                              "kind": "COUNTER",
                              "units": "BYTES"
                          },
                          {
                              "name": "impala-server.io-mgr.bytes-written",
                              "description": "Total number of bytes written to disk by the IO manager.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "BYTES"
                          },
                          {
                              "name": "impala-server.io-mgr.cached-bytes-read",
                              "description": "Total number of cached bytes read by the IO manager.",
                              "human_readable": "15.53 KB",
                              "value": 15898,
                              "kind": "COUNTER",
                              "units": "BYTES"
                          },
                          {
                              "name": "impala-server.io-mgr.local-bytes-read",
                              "description": "Total number of local bytes read by the IO manager.",
                              "human_readable": "4.52 GB",
                              "value": 4852715212,
                              "kind": "COUNTER",
                              "units": "BYTES"
                          },
                          {
                              "name": "impala-server.io-mgr.num-open-files",
                              "description": "The current number of files opened by the IO Manager",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.io-mgr.short-circuit-bytes-read",
                              "description": "Total number of short-circuit bytes read by the IO manager.",
                              "human_readable": "4.52 GB",
                              "value": 4852715212,
                              "kind": "COUNTER",
                              "units": "BYTES"
                          },
                          {
                              "name": "impala-server.io.mgr.cached-file-handles-hit-count",
                              "description": "Number of cache hits for cached HDFS file handles",
                              "human_readable": "10397",
                              "value": 10397,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.io.mgr.cached-file-handles-hit-ratio",
                              "description": "HDFS file handle cache hit ratio, between 0 and 1, where 1 means all reads were served from cached file handles.",
                              "human_readable": "last: 1, mean: 0.737428",
                              "units": "NONE",
                              "last": 1,
                              "mean": 0.737428
                          },
                          {
                              "name": "impala-server.io.mgr.cached-file-handles-miss-count",
                              "description": "Number of cache misses for cached HDFS file handles",
                              "human_readable": "3702",
                              "value": 3702,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.io.mgr.cached-file-handles-reopened",
                              "description": "Number of cached HDFS file handles reopened",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.io.mgr.num-cached-file-handles",
                              "description": "Number of currently cached HDFS file handles in the IO manager.",
                              "human_readable": "1082",
                              "value": 1082,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.io.mgr.num-file-handles-outstanding",
                              "description": "Number of HDFS file handles that are currently in use by readers.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.mem-pool.total-bytes",
                              "description": "The current size of the memory pool shared by all queries",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "impala-server.num-files-open-for-insert",
                              "description": "The number of HDFS files currently open for writing.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.num-fragments",
                              "description": "The total number of query fragments processed over the life of the process.",
                              "human_readable": "4.30K",
                              "value": 4298,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "impala-server.num-fragments-in-flight",
                              "description": "The number of query fragments currently executing.",
                              "human_readable": "1",
                              "value": 1,
                              "kind": "GAUGE",
                              "units": "UNIT"
                          },
                          {
                              "name": "impala-server.num-open-beeswax-sessions",
                              "description": "The number of open Beeswax sessions.",
                              "human_readable": "1",
                              "value": 1,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.num-open-hiveserver2-sessions",
                              "description": "The number of open HiveServer2 sessions.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.num-queries",
                              "description": "The total number of queries processed over the life of the process",
                              "human_readable": "4.17K",
                              "value": 4171,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "impala-server.num-queries-expired",
                              "description": "Number of queries expired due to inactivity.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "impala-server.num-queries-registered",
                              "description": "The total number of queries registered on this Impala server instance. Includes queries that are in flight and waiting to be closed",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "UNIT"
                          },
                          {
                              "name": "impala-server.num-queries-spilled",
                              "description": "Number of queries for which any operator spilled.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "impala-server.num-sessions-expired",
                              "description": "Number of sessions expired due to inactivity.",
                              "human_readable": "4",
                              "value": 4,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "impala-server.query-durations-ms",
                              "description": "Distribution of query latencies",
                              "human_readable": "Count: 2029, min / max: 1ms / 27s134ms, 25th %-ile: 124ms, 50th %-ile: 339ms, 75th %-ile: 1s019ms, 90th %-ile: 4s600ms, 95th %-ile: 5s884ms, 99.9th %-ile: 22s912ms",
                              "25th %-ile": 124,
                              "50th %-ile": 339,
                              "75th %-ile": 1019,
                              "90th %-ile": 4600,
                              "95th %-ile": 5884,
                              "99.9th %-ile": 22912,
                              "max": 27134,
                              "min": 1,
                              "count": 2029,
                              "kind": "HISTOGRAM",
                              "units": "TIME_MS"
                          },
                          {
                              "name": "impala-server.ready",
                              "description": "Indicates if the Impala Server is ready.",
                              "human_readable": "true",
                              "value": true,
                              "kind": "PROPERTY",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.resultset-cache.total-bytes",
                              "description": "Total number of bytes consumed for rows cached to support HS2 FETCH_FIRST.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.resultset-cache.total-num-rows",
                              "description": "Total number of rows cached to support HS2 FETCH_FIRST.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "impala-server.scan-ranges.num-missing-volume-id",
                              "description": "The total number of scan ranges read over the life of the process that did not have volume metadata",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "impala-server.scan-ranges.total",
                              "description": "The total number of scan ranges read over the life of the process",
                              "human_readable": "5.48K",
                              "value": 5482,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "impala-server.version",
                              "description": "The full version string of the Impala Server.",
                              "human_readable": "impalad version 2.13.0-SNAPSHOT DEBUG (build 8d6142fd2ec00b095d9a55ca9bba76d0d43d3ab3)",
                              "value": "impalad version 2.13.0-SNAPSHOT DEBUG (build 8d6142fd2ec00b095d9a55ca9bba76d0d43d3ab3)",
                              "kind": "PROPERTY",
                              "units": "NONE"
                          }
                      ],
                      "name": "impala-server",
                      "child_groups": [
                          {
                              "metrics": [
                                  {
                                      "name": "catalog.curr-serviceid",
                                      "description": "Catalog service id.",
                                      "human_readable": "29902ad192db4205:95293c2e587cc087",
                                      "value": "29902ad192db4205:95293c2e587cc087",
                                      "kind": "PROPERTY",
                                      "units": "NONE"
                                  },
                                  {
                                      "name": "catalog.curr-topic",
                                      "description": "Statestore topic update version.",
                                      "human_readable": "32532",
                                      "value": 32532,
                                      "kind": "GAUGE",
                                      "units": "NONE"
                                  },
                                  {
                                      "name": "catalog.curr-version",
                                      "description": "Catalog topic update version.",
                                      "human_readable": "32271",
                                      "value": 32271,
                                      "kind": "GAUGE",
                                      "units": "NONE"
                                  },
                                  {
                                      "name": "catalog.num-databases",
                                      "description": "The number of databases in the catalog.",
                                      "human_readable": "45",
                                      "value": 45,
                                      "kind": "GAUGE",
                                      "units": "NONE"
                                  },
                                  {
                                      "name": "catalog.num-tables",
                                      "description": "The number of tables in the catalog.",
                                      "human_readable": "1090",
                                      "value": 1090,
                                      "kind": "GAUGE",
                                      "units": "NONE"
                                  },
                                  {
                                      "name": "catalog.ready",
                                      "description": "Indicates if the catalog is ready.",
                                      "human_readable": "true",
                                      "value": true,
                                      "kind": "PROPERTY",
                                      "units": "NONE"
                                  }
                              ],
                              "name": "catalog",
                              "child_groups": []
                          }
                      ]
                  },
                  {
                      "metrics": [
                          {
                              "name": "jvm.code-cache.committed-usage-bytes",
                              "description": "Jvm code-cache Committed Usage Bytes",
                              "human_readable": "41.50 MB",
                              "value": 43515904,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.code-cache.current-usage-bytes",
                              "description": "Jvm code-cache Current Usage Bytes",
                              "human_readable": "41.19 MB",
                              "value": 43189696,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.code-cache.init-usage-bytes",
                              "description": "Jvm code-cache Init Usage Bytes",
                              "human_readable": "2.44 MB",
                              "value": 2555904,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.code-cache.max-usage-bytes",
                              "description": "Jvm code-cache Max Usage Bytes",
                              "human_readable": "240.00 MB",
                              "value": 251658240,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.code-cache.peak-committed-usage-bytes",
                              "description": "Jvm code-cache Peak Committed Usage Bytes",
                              "human_readable": "41.50 MB",
                              "value": 43515904,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.code-cache.peak-current-usage-bytes",
                              "description": "Jvm code-cache Peak Current Usage Bytes",
                              "human_readable": "41.20 MB",
                              "value": 43200256,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.code-cache.peak-init-usage-bytes",
                              "description": "Jvm code-cache Peak Init Usage Bytes",
                              "human_readable": "2.44 MB",
                              "value": 2555904,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.code-cache.peak-max-usage-bytes",
                              "description": "Jvm code-cache Peak Max Usage Bytes",
                              "human_readable": "240.00 MB",
                              "value": 251658240,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.compressed-class-space.committed-usage-bytes",
                              "description": "Jvm compressed-class-space Committed Usage Bytes",
                              "human_readable": "5.88 MB",
                              "value": 6160384,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.compressed-class-space.current-usage-bytes",
                              "description": "Jvm compressed-class-space Current Usage Bytes",
                              "human_readable": "5.77 MB",
                              "value": 6050168,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.compressed-class-space.init-usage-bytes",
                              "description": "Jvm compressed-class-space Init Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.compressed-class-space.max-usage-bytes",
                              "description": "Jvm compressed-class-space Max Usage Bytes",
                              "human_readable": "1.00 GB",
                              "value": 1073741824,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.compressed-class-space.peak-committed-usage-bytes",
                              "description": "Jvm compressed-class-space Peak Committed Usage Bytes",
                              "human_readable": "5.88 MB",
                              "value": 6160384,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.compressed-class-space.peak-current-usage-bytes",
                              "description": "Jvm compressed-class-space Peak Current Usage Bytes",
                              "human_readable": "5.77 MB",
                              "value": 6050168,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.compressed-class-space.peak-init-usage-bytes",
                              "description": "Jvm compressed-class-space Peak Init Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.compressed-class-space.peak-max-usage-bytes",
                              "description": "Jvm compressed-class-space Peak Max Usage Bytes",
                              "human_readable": "1.00 GB",
                              "value": 1073741824,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.heap.committed-usage-bytes",
                              "description": "Jvm heap Committed Usage Bytes",
                              "human_readable": "97.50 MB",
                              "value": 102236160,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.heap.current-usage-bytes",
                              "description": "Jvm heap Current Usage Bytes",
                              "human_readable": "84.55 MB",
                              "value": 88679024,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.heap.init-usage-bytes",
                              "description": "Jvm heap Init Usage Bytes",
                              "human_readable": "1.04 GB",
                              "value": 1111490560,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.heap.max-usage-bytes",
                              "description": "Jvm heap Max Usage Bytes",
                              "human_readable": "14.72 GB",
                              "value": 15802564608,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.heap.peak-committed-usage-bytes",
                              "description": "Jvm heap Peak Committed Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.heap.peak-current-usage-bytes",
                              "description": "Jvm heap Peak Current Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.heap.peak-init-usage-bytes",
                              "description": "Jvm heap Peak Init Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.heap.peak-max-usage-bytes",
                              "description": "Jvm heap Peak Max Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.metaspace.committed-usage-bytes",
                              "description": "Jvm metaspace Committed Usage Bytes",
                              "human_readable": "58.88 MB",
                              "value": 61734912,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.metaspace.current-usage-bytes",
                              "description": "Jvm metaspace Current Usage Bytes",
                              "human_readable": "58.35 MB",
                              "value": 61187992,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.metaspace.init-usage-bytes",
                              "description": "Jvm metaspace Init Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.metaspace.max-usage-bytes",
                              "description": "Jvm metaspace Max Usage Bytes",
                              "human_readable": "-1.00 B",
                              "value": -1,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.metaspace.peak-committed-usage-bytes",
                              "description": "Jvm metaspace Peak Committed Usage Bytes",
                              "human_readable": "58.88 MB",
                              "value": 61734912,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.metaspace.peak-current-usage-bytes",
                              "description": "Jvm metaspace Peak Current Usage Bytes",
                              "human_readable": "58.35 MB",
                              "value": 61187992,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.metaspace.peak-init-usage-bytes",
                              "description": "Jvm metaspace Peak Init Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.metaspace.peak-max-usage-bytes",
                              "description": "Jvm metaspace Peak Max Usage Bytes",
                              "human_readable": "-1.00 B",
                              "value": -1,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.non-heap.committed-usage-bytes",
                              "description": "Jvm non-heap Committed Usage Bytes",
                              "human_readable": "106.25 MB",
                              "value": 111411200,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.non-heap.current-usage-bytes",
                              "description": "Jvm non-heap Current Usage Bytes",
                              "human_readable": "105.31 MB",
                              "value": 110427856,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.non-heap.init-usage-bytes",
                              "description": "Jvm non-heap Init Usage Bytes",
                              "human_readable": "2.44 MB",
                              "value": 2555904,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.non-heap.max-usage-bytes",
                              "description": "Jvm non-heap Max Usage Bytes",
                              "human_readable": "-1.00 B",
                              "value": -1,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.non-heap.peak-committed-usage-bytes",
                              "description": "Jvm non-heap Peak Committed Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.non-heap.peak-current-usage-bytes",
                              "description": "Jvm non-heap Peak Current Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.non-heap.peak-init-usage-bytes",
                              "description": "Jvm non-heap Peak Init Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.non-heap.peak-max-usage-bytes",
                              "description": "Jvm non-heap Peak Max Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-eden-space.committed-usage-bytes",
                              "description": "Jvm ps-eden-space Committed Usage Bytes",
                              "human_readable": "43.00 MB",
                              "value": 45088768,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-eden-space.current-usage-bytes",
                              "description": "Jvm ps-eden-space Current Usage Bytes",
                              "human_readable": "43.00 MB",
                              "value": 45088768,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-eden-space.init-usage-bytes",
                              "description": "Jvm ps-eden-space Init Usage Bytes",
                              "human_readable": "512.00 KB",
                              "value": 524288,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-eden-space.max-usage-bytes",
                              "description": "Jvm ps-eden-space Max Usage Bytes",
                              "human_readable": "5.50 GB",
                              "value": 5905055744,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-eden-space.peak-committed-usage-bytes",
                              "description": "Jvm ps-eden-space Peak Committed Usage Bytes",
                              "human_readable": "78.50 MB",
                              "value": 82313216,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-eden-space.peak-current-usage-bytes",
                              "description": "Jvm ps-eden-space Peak Current Usage Bytes",
                              "human_readable": "78.50 MB",
                              "value": 82313216,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-eden-space.peak-init-usage-bytes",
                              "description": "Jvm ps-eden-space Peak Init Usage Bytes",
                              "human_readable": "512.00 KB",
                              "value": 524288,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-eden-space.peak-max-usage-bytes",
                              "description": "Jvm ps-eden-space Peak Max Usage Bytes",
                              "human_readable": "5.52 GB",
                              "value": 5924454400,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-old-gen.committed-usage-bytes",
                              "description": "Jvm ps-old-gen Committed Usage Bytes",
                              "human_readable": "43.50 MB",
                              "value": 45613056,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-old-gen.current-usage-bytes",
                              "description": "Jvm ps-old-gen Current Usage Bytes",
                              "human_readable": "37.45 MB",
                              "value": 39273160,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-old-gen.init-usage-bytes",
                              "description": "Jvm ps-old-gen Init Usage Bytes",
                              "human_readable": "1.03 GB",
                              "value": 1109917696,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-old-gen.max-usage-bytes",
                              "description": "Jvm ps-old-gen Max Usage Bytes",
                              "human_readable": "11.04 GB",
                              "value": 11852054528,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-old-gen.peak-committed-usage-bytes",
                              "description": "Jvm ps-old-gen Peak Committed Usage Bytes",
                              "human_readable": "1.03 GB",
                              "value": 1109917696,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-old-gen.peak-current-usage-bytes",
                              "description": "Jvm ps-old-gen Peak Current Usage Bytes",
                              "human_readable": "37.45 MB",
                              "value": 39273160,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-old-gen.peak-init-usage-bytes",
                              "description": "Jvm ps-old-gen Peak Init Usage Bytes",
                              "human_readable": "1.03 GB",
                              "value": 1109917696,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-old-gen.peak-max-usage-bytes",
                              "description": "Jvm ps-old-gen Peak Max Usage Bytes",
                              "human_readable": "11.04 GB",
                              "value": 11852054528,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-survivor-space.committed-usage-bytes",
                              "description": "Jvm ps-survivor-space Committed Usage Bytes",
                              "human_readable": "9.50 MB",
                              "value": 9961472,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-survivor-space.current-usage-bytes",
                              "description": "Jvm ps-survivor-space Current Usage Bytes",
                              "human_readable": "3.03 MB",
                              "value": 3178496,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-survivor-space.init-usage-bytes",
                              "description": "Jvm ps-survivor-space Init Usage Bytes",
                              "human_readable": "512.00 KB",
                              "value": 524288,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-survivor-space.max-usage-bytes",
                              "description": "Jvm ps-survivor-space Max Usage Bytes",
                              "human_readable": "9.50 MB",
                              "value": 9961472,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-survivor-space.peak-committed-usage-bytes",
                              "description": "Jvm ps-survivor-space Peak Committed Usage Bytes",
                              "human_readable": "11.50 MB",
                              "value": 12058624,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-survivor-space.peak-current-usage-bytes",
                              "description": "Jvm ps-survivor-space Peak Current Usage Bytes",
                              "human_readable": "9.94 MB",
                              "value": 10420240,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-survivor-space.peak-init-usage-bytes",
                              "description": "Jvm ps-survivor-space Peak Init Usage Bytes",
                              "human_readable": "512.00 KB",
                              "value": 524288,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.ps-survivor-space.peak-max-usage-bytes",
                              "description": "Jvm ps-survivor-space Peak Max Usage Bytes",
                              "human_readable": "11.50 MB",
                              "value": 12058624,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.total.committed-usage-bytes",
                              "description": "Jvm total Committed Usage Bytes",
                              "human_readable": "203.25 MB",
                              "value": 213123072,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.total.current-usage-bytes",
                              "description": "Jvm total Current Usage Bytes",
                              "human_readable": "145.89 MB",
                              "value": 152973824,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.total.init-usage-bytes",
                              "description": "Jvm total Init Usage Bytes",
                              "human_readable": "1.04 GB",
                              "value": 1113522176,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.total.max-usage-bytes",
                              "description": "Jvm total Max Usage Bytes",
                              "human_readable": "17.78 GB",
                              "value": 19092471807,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.total.peak-committed-usage-bytes",
                              "description": "Jvm total Peak Committed Usage Bytes",
                              "human_readable": "1.23 GB",
                              "value": 1315700736,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.total.peak-current-usage-bytes",
                              "description": "Jvm total Peak Current Usage Bytes",
                              "human_readable": "231.21 MB",
                              "value": 242445032,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.total.peak-init-usage-bytes",
                              "description": "Jvm total Peak Init Usage Bytes",
                              "human_readable": "1.04 GB",
                              "value": 1113522176,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "jvm.total.peak-max-usage-bytes",
                              "description": "Jvm total Peak Max Usage Bytes",
                              "human_readable": "17.80 GB",
                              "value": 19113967615,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          }
                      ],
                      "name": "jvm",
                      "child_groups": []
                  },
                  {
                      "metrics": [
                          {
                              "name": "memory.anon-huge-page-bytes",
                              "description": "Total bytes of anonymous (a.k.a. transparent) huge pages used by this process.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "memory.mapped-bytes",
                              "description": "Total bytes of memory mappings in this process (the virtual memory size).",
                              "human_readable": "28.26 GB",
                              "value": 30344564736,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "memory.num-maps",
                              "description": "Total number of memory mappings in this process.",
                              "human_readable": "989",
                              "value": 989,
                              "kind": "GAUGE",
                              "units": "NONE"
                          },
                          {
                              "name": "memory.rss",
                              "description": "Resident set size (RSS) of this process, including TCMalloc, buffer pool and Jvm.",
                              "human_readable": "1.46 GB",
                              "value": 1567739904,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "memory.thp.defrag",
                              "description": "The system-wide 'defrag' setting for Transparent Huge Pages.",
                              "human_readable": "<unknown>",
                              "value": "<unknown>",
                              "kind": "PROPERTY",
                              "units": "NONE"
                          },
                          {
                              "name": "memory.thp.enabled",
                              "description": "The system-wide 'enabled' setting for Transparent Huge Pages.",
                              "human_readable": "<unknown>",
                              "value": "<unknown>",
                              "kind": "PROPERTY",
                              "units": "NONE"
                          },
                          {
                              "name": "memory.thp.khugepaged-defrag",
                              "description": "The system-wide 'defrag' setting for khugepaged.",
                              "human_readable": "<unknown>",
                              "value": "<unknown>",
                              "kind": "PROPERTY",
                              "units": "NONE"
                          },
                          {
                              "name": "memory.total-used",
                              "description": "Total memory currently used by TCMalloc and buffer pool.",
                              "human_readable": "890.26 MB",
                              "value": 933502976,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          }
                      ],
                      "name": "memory",
                      "child_groups": []
                  },
                  {
                      "metrics": [
                          {
                              "name": "mem-tracker.DataStreamService.current_usage_bytes",
                              "description": "Memtracker DataStreamService Current Usage Bytes",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "mem-tracker.DataStreamService.peak_usage_bytes",
                              "description": "Memtracker DataStreamService Peak Usage Bytes",
                              "human_readable": "77.29 KB",
                              "value": 79142,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "rpc.impala.DataStreamService.rpcs_queue_overflow",
                              "description": "Service impala.DataStreamService: Total number of incoming RPCs that were rejected due to overflow of the service queue.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          }
                      ],
                      "name": "rpc",
                      "child_groups": []
                  },
                  {
                      "metrics": [
                          {
                              "name": "simple-scheduler.assignments.total",
                              "description": "The number of assignments",
                              "human_readable": "12.99K",
                              "value": 12994,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "simple-scheduler.initialized",
                              "description": "Indicates whether the scheduler has been initialized.",
                              "human_readable": "true",
                              "value": true,
                              "kind": "PROPERTY",
                              "units": "NONE"
                          },
                          {
                              "name": "simple-scheduler.local-assignments.total",
                              "description": "Number of assignments operating on local data",
                              "human_readable": "12.80K",
                              "value": 12805,
                              "kind": "COUNTER",
                              "units": "UNIT"
                          },
                          {
                              "name": "simple-scheduler.num-backends",
                              "description": "The number of backend connections from this Impala Daemon to other Impala Daemons.",
                              "human_readable": "3",
                              "value": 3,
                              "kind": "GAUGE",
                              "units": "NONE"
                          }
                      ],
                      "name": "scheduler",
                      "child_groups": []
                  },
                  {
                      "metrics": [
                          {
                              "name": "rpc-method.statestore-subscriber.StatestoreSubscriber.Heartbeat.call_duration",
                              "description": "",
                              "human_readable": "Count: 10924, min / max: 0 / 1ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                              "25th %-ile": 0,
                              "50th %-ile": 0,
                              "75th %-ile": 0,
                              "90th %-ile": 0,
                              "95th %-ile": 1,
                              "99.9th %-ile": 1,
                              "max": 1,
                              "min": 0,
                              "count": 10924,
                              "kind": "HISTOGRAM",
                              "units": "TIME_MS"
                          },
                          {
                              "name": "rpc-method.statestore-subscriber.StatestoreSubscriber.UpdateState.call_duration",
                              "description": "",
                              "human_readable": "Count: 114070, min / max: 0 / 687ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 7ms",
                              "25th %-ile": 0,
                              "50th %-ile": 0,
                              "75th %-ile": 0,
                              "90th %-ile": 1,
                              "95th %-ile": 1,
                              "99.9th %-ile": 7,
                              "max": 687,
                              "min": 0,
                              "count": 114070,
                              "kind": "HISTOGRAM",
                              "units": "TIME_MS"
                          },
                          {
                              "name": "statestore-subscriber.connected",
                              "description": "Whether the Impala Daemon considers itself connected to the StateStore.",
                              "human_readable": "true",
                              "value": true,
                              "kind": "PROPERTY",
                              "units": "NONE"
                          },
                          {
                              "name": "statestore-subscriber.heartbeat-interval-time",
                              "description": "The time (sec) between Statestore heartbeats.",
                              "human_readable": "count: 10924, last: 999ms, min: 2ms, max: 1s004ms, mean: 1s000ms, stddev: 9ms",
                              "units": "TIME_S",
                              "count": 10924,
                              "last": 0.999994,
                              "min": 0,
                              "max": 1.00499,
                              "mean": 1.00038,
                              "stddev": 0.00956347
                          },
                          {
                              "name": "statestore-subscriber.last-recovery-duration",
                              "description": "The amount of time the StateStore subscriber took to recover the connection the last time it was lost.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "TIME_S"
                          },
                          {
                              "name": "statestore-subscriber.last-recovery-time",
                              "description": "The local time that the last statestore recovery happened.",
                              "human_readable": "N/A",
                              "value": "N/A",
                              "kind": "PROPERTY",
                              "units": "NONE"
                          },
                          {
                              "name": "statestore-subscriber.topic-catalog-update.processing-time-s",
                              "description": "Statestore Subscriber Topic catalog-update Processing Time",
                              "human_readable": "count: 5455, last: 2ms, min: 0, max: 685ms, mean: 3ms, stddev: 15ms",
                              "units": "TIME_S",
                              "count": 5455,
                              "last": 0.00299998,
                              "min": 0,
                              "max": 0.685996,
                              "mean": 0.00301336,
                              "stddev": 0.0152849
                          },
                          {
                              "name": "statestore-subscriber.topic-catalog-update.update-interval",
                              "description": "Interval between topic updates for Topic catalog-update",
                              "human_readable": "count: 5455, last: 2s000ms, min: 40ms, max: 2s006ms, mean: 2s000ms, stddev: 26ms",
                              "units": "TIME_S",
                              "count": 5455,
                              "last": 2.00099,
                              "min": 0,
                              "max": 2.00699,
                              "mean": 2.00022,
                              "stddev": 0.0265383
                          },
                          {
                              "name": "statestore-subscriber.topic-impala-membership.processing-time-s",
                              "description": "Statestore Subscriber Topic impala-membership Processing Time",
                              "human_readable": "count: 108615, last: 0, min: 0, max: 14ms, mean: 0ms, stddev: 0ms",
                              "units": "TIME_S",
                              "count": 108615,
                              "last": 0,
                              "min": 0,
                              "max": 0.0149999,
                              "mean": 1.52648e-05,
                              "stddev": 0.000133393
                          },
                          {
                              "name": "statestore-subscriber.topic-impala-membership.update-interval",
                              "description": "Interval between topic updates for Topic impala-membership",
                              "human_readable": "count: 108615, last: 100ms, min: 90ms, max: 338ms, mean: 100ms, stddev: 0ms",
                              "units": "TIME_S",
                              "count": 108615,
                              "last": 0.100999,
                              "min": 0,
                              "max": 0.338998,
                              "mean": 0.10054,
                              "stddev": 0.000941269
                          },
                          {
                              "name": "statestore-subscriber.topic-impala-request-queue.processing-time-s",
                              "description": "Statestore Subscriber Topic impala-request-queue Processing Time",
                              "human_readable": "count: 108615, last: 0, min: 0, max: 14ms, mean: 0ms, stddev: 0ms",
                              "units": "TIME_S",
                              "count": 108615,
                              "last": 0,
                              "min": 0,
                              "max": 0.0149999,
                              "mean": 6.63164e-05,
                              "stddev": 0.000261428
                          },
                          {
                              "name": "statestore-subscriber.topic-impala-request-queue.update-interval",
                              "description": "Interval between topic updates for Topic impala-request-queue",
                              "human_readable": "count: 108615, last: 100ms, min: 90ms, max: 338ms, mean: 100ms, stddev: 0ms",
                              "units": "TIME_S",
                              "count": 108615,
                              "last": 0.100999,
                              "min": 0,
                              "max": 0.338998,
                              "mean": 0.100543,
                              "stddev": 0.0009398
                          },
                          {
                              "name": "statestore-subscriber.topic-update-duration",
                              "description": "The time (sec) taken to process Statestore subscriber topic updates.",
                              "human_readable": "count: 114070, last: 0, min: 0, max: 685ms, mean: 0ms, stddev: 3ms",
                              "units": "TIME_S",
                              "count": 114070,
                              "last": 0,
                              "min": 0,
                              "max": 0.685996,
                              "mean": 0.0002179,
                              "stddev": 0.00341233
                          },
                          {
                              "name": "statestore-subscriber.topic-update-interval-time",
                              "description": "The time (sec) between Statestore subscriber topic updates.",
                              "human_readable": "count: 222685, last: 100ms, min: 40ms, max: 2s006ms, mean: 147ms, stddev: 293ms",
                              "units": "TIME_S",
                              "count": 222685,
                              "last": 0.100999,
                              "min": 0,
                              "max": 2.00699,
                              "mean": 0.147077,
                              "stddev": 0.293692
                          }
                      ],
                      "name": "statestore-subscriber",
                      "child_groups": []
                  },
                  {
                      "metrics": [
                          {
                              "name": "tcmalloc.bytes-in-use",
                              "description": "Number of bytes used by the application. This will not typically match the memory use reported by the OS, because it does not include TCMalloc overhead or memory fragmentation.",
                              "human_readable": "721.56 MB",
                              "value": 754242136,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "tcmalloc.pageheap-free-bytes",
                              "description": "Number of bytes in free, mapped pages in page heap. These bytes can be used to fulfill allocation requests. They always count towards virtual memory usage, and unless the underlying memory is swapped out by the OS, they also count towards physical memory usage.",
                              "human_readable": "0",
                              "value": 0,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "tcmalloc.pageheap-unmapped-bytes",
                              "description": "Number of bytes in free, unmapped pages in page heap. These are bytes that have been released back to the OS, possibly by one of the MallocExtension \"Release\" calls. They can be used to fulfill allocation requests, but typically incur a page fault. They always count towards virtual memory usage, and depending on the OS, typically do not count towards physical memory usage.",
                              "human_readable": "6.78 GB",
                              "value": 7284998144,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "tcmalloc.physical-bytes-reserved",
                              "description": "Derived metric computing the amount of physical memory (in bytes) used by the process, including that actually in use and free bytes reserved by tcmalloc. Does not include the tcmalloc metadata.",
                              "human_readable": "875.30 MB",
                              "value": 917823488,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          },
                          {
                              "name": "tcmalloc.total-bytes-reserved",
                              "description": "Bytes of system memory reserved by TCMalloc.",
                              "human_readable": "7.64 GB",
                              "value": 8202821632,
                              "kind": "GAUGE",
                              "units": "BYTES"
                          }
                      ],
                      "name": "tcmalloc",
                      "child_groups": []
                  }
              ]
          }
      }"
      13:31:54 E   queries: "{
          "__common__": {
              "process-name": "impalad",
              "navbar": [
                  {
                      "link": "/",
                      "title": "/"
                  },
                  {
                      "link": "/backends",
                      "title": "/backends"
                  },
                  {
                      "link": "/catalog",
                      "title": "/catalog"
                  },
                  {
                      "link": "/hadoop-varz",
                      "title": "/hadoop-varz"
                  },
                  {
                      "link": "/log_level",
                      "title": "/log_level"
                  },
                  {
                      "link": "/logs",
                      "title": "/logs"
                  },
                  {
                      "link": "/memz",
                      "title": "/memz"
                  },
                  {
                      "link": "/metrics",
                      "title": "/metrics"
                  },
                  {
                      "link": "/queries",
                      "title": "/queries"
                  },
                  {
                      "link": "/rpcz",
                      "title": "/rpcz"
                  },
                  {
                      "link": "/sessions",
                      "title": "/sessions"
                  },
                  {
                      "link": "/threadz",
                      "title": "/threadz"
                  },
                  {
                      "link": "/varz",
                      "title": "/varz"
                  }
              ]
          },
          "in_flight_queries": [],
          "num_in_flight_queries": 0,
          "num_executing_queries": 0,
          "num_waiting_queries": 0,
          "waiting-tooltip": "These queries are no longer executing, either because they encountered an error or because they have returned all of their results, but they are still active so that their results can be inspected. To free the resources they are using, they must be closed.",
          "completed_queries": [
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "invalidate metadata functional_parquet. alltypesinsert",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:41:53.397556000",
                  "end_time": "2018-06-13 12:41:53.416333000",
                  "duration": "18.777ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "f4473857011f5e57:f8061d6a00000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "1m2s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "default",
                  "stmt": "use functional_parquet",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:41:53.357679000",
                  "end_time": "2018-06-13 12:41:53.363695000",
                  "duration": "6.016ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "7b41c1fa684c29db:9e4af83f00000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "1m2s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "SET MEM_LIMIT=\"0\"",
                  "stmt_type": "SET",
                  "start_time": "2018-06-13 12:41:03.589021000",
                  "end_time": "2018-06-13 12:41:03.594748000",
                  "duration": "5.727ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "d14a0fa71d2e57a1:8a1ead1700000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "1m51s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "insert into table alltypesinsert\npartition (year, month) /* +noclustered */\nselect at1.id, at1.bool_col, at1.tinyint_col, at1.smallint_col, at1.int_col, at1.bigint_col,\n  at1.float_col, at1.double_col, at1.date_string_col, at1.string_col, at1.timestamp_col,\n  at1.year, at2.id as month\nfrom  functional.alltypes at1, functional.alltypes at2",
                  "stmt_type": "DML",
                  "start_time": "2018-06-13 12:40:59.857990000",
                  "end_time": "2018-06-13 12:41:03.569236000",
                  "duration": "3s711ms",
                  "progress": "40 / 48 (83.3333%)",
                  "state": "EXCEPTION",
                  "rows_fetched": 0,
                  "query_id": "d74c601d73335591:769d4f2500000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "1m51s",
                  "resource_pool": "default-pool"
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "set mem_limit=64m",
                  "stmt_type": "SET",
                  "start_time": "2018-06-13 12:40:59.848447000",
                  "end_time": "2018-06-13 12:40:59.852279000",
                  "duration": "3.832ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "14cd28a8d3c1152:e83c0fd700000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "1m55s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "invalidate metadata functional_parquet. alltypesinsert",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:40:59.823319000",
                  "end_time": "2018-06-13 12:40:59.842609000",
                  "duration": "19.290ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "d540cd5d4461f76a:58674dd300000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "1m55s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional",
                  "stmt": "use functional_parquet",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:40:59.809252000",
                  "end_time": "2018-06-13 12:40:59.814263000",
                  "duration": "5.011ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "e740d34318828531:c837811700000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "1m55s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "SET MEM_LIMIT=\"0\"",
                  "stmt_type": "SET",
                  "start_time": "2018-06-13 12:40:45.489818000",
                  "end_time": "2018-06-13 12:40:45.494902000",
                  "duration": "5.084ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "9f4c12a325dfce5a:cc12e9ee00000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "2m10s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "insert into table alltypesinsert\npartition (year, month) /* +noclustered */\nselect at1.id, at1.bool_col, at1.tinyint_col, at1.smallint_col, at1.int_col, at1.bigint_col,\n  at1.float_col, at1.double_col, at1.date_string_col, at1.string_col, at1.timestamp_col,\n  at1.year, at2.id as month\nfrom  functional.alltypes at1, functional.alltypes at2",
                  "stmt_type": "DML",
                  "start_time": "2018-06-13 12:40:31.520697000",
                  "end_time": "2018-06-13 12:40:45.448965000",
                  "duration": "13s928ms",
                  "progress": "48 / 48 ( 100%)",
                  "state": "EXCEPTION",
                  "rows_fetched": 0,
                  "query_id": "854c5a2b234c833f:6fe7f01f00000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "2m10s",
                  "resource_pool": "default-pool"
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "set mem_limit=64m",
                  "stmt_type": "SET",
                  "start_time": "2018-06-13 12:40:31.510422000",
                  "end_time": "2018-06-13 12:40:31.514444000",
                  "duration": "4.022ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "d044f22b071bb8f7:7588696400000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "2m24s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional",
                  "stmt": "invalidate metadata functional_parquet. alltypesinsert",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:40:31.485279000",
                  "end_time": "2018-06-13 12:40:31.504565000",
                  "duration": "19.286ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "cf4bb6ae26a2bdb2:d1befa5800000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "2m24s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "default",
                  "stmt": "use functional_parquet",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:40:31.139409000",
                  "end_time": "2018-06-13 12:40:31.145522000",
                  "duration": "6.113ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "294ac3123c895940:9e7c75e700000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "2m24s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional",
                  "stmt": "SET MEM_LIMIT=\"0\"",
                  "stmt_type": "SET",
                  "start_time": "2018-06-13 12:40:21.817754000",
                  "end_time": "2018-06-13 12:40:21.873564000",
                  "duration": "55.810ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "e84dae5ae6ebdeb0:5d389bc800000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "2m33s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional",
                  "stmt": "insert into table alltypesinsert\npartition (year, month) /* +noclustered */\nselect at1.id, at1.bool_col, at1.tinyint_col, at1.smallint_col, at1.int_col, at1.bigint_col,\n  at1.float_col, at1.double_col, at1.date_string_col, at1.string_col, at1.timestamp_col,\n  at1.year, at2.id as month\nfrom  functional.alltypes at1, functional.alltypes at2",
                  "stmt_type": "DML",
                  "start_time": "2018-06-13 12:40:11.462182000",
                  "end_time": "2018-06-13 12:40:21.777134000",
                  "duration": "10s314ms",
                  "progress": "48 / 48 ( 100%)",
                  "state": "EXCEPTION",
                  "rows_fetched": 0,
                  "query_id": "aa4ca03c1764eae8:9304c90200000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "2m33s",
                  "resource_pool": "default-pool"
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional",
                  "stmt": "set mem_limit=64m",
                  "stmt_type": "SET",
                  "start_time": "2018-06-13 12:40:11.451993000",
                  "end_time": "2018-06-13 12:40:11.456193000",
                  "duration": "4.200ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "774c497c535fc138:8bbeb97100000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "2m44s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional",
                  "stmt": "invalidate metadata functional. alltypesinsert",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:40:11.426945000",
                  "end_time": "2018-06-13 12:40:11.446505000",
                  "duration": "19.560ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "ed4af87497729284:a2f7306700000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "2m44s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "use functional",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:40:11.032209000",
                  "end_time": "2018-06-13 12:40:11.037541000",
                  "duration": "5.332ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "644da258ebb960ad:6b7bca9900000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "2m44s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "insert into table alltypesnopart_insert\n /*+ clustered,shuffle */\nselect id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col,\ndouble_col, date_string_col, string_col, timestamp_col from alltypessmall",
                  "stmt_type": "DML",
                  "start_time": "2018-06-13 12:39:55.079086000",
                  "end_time": "2018-06-13 12:40:04.319447000",
                  "duration": "9s240ms",
                  "progress": "4 / 4 ( 100%)",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "6c4ec6a3bd6244c8:becf850300000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "2m51s",
                  "resource_pool": "default-pool"
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "invalidate metadata functional_parquet. alltypesnopart_insert",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:39:55.053221000",
                  "end_time": "2018-06-13 12:39:55.073433000",
                  "duration": "20.212ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "b14d3d074c3db943:4832101b00000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "3m",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "invalidate metadata functional_parquet. alltypesinsert",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:39:49.311880000",
                  "end_time": "2018-06-13 12:39:49.364360000",
                  "duration": "52.480ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "8e49a44962e85b00:46025e4400000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "3m6s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "invalidate metadata functional_parquet. alltypesinsert",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:39:49.205368000",
                  "end_time": "2018-06-13 12:39:49.224240000",
                  "duration": "18.872ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "24207828837d7e8:3339287600000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "3m6s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "invalidate metadata functional_parquet. alltypesinsert",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:39:42.849499000",
                  "end_time": "2018-06-13 12:39:42.870306000",
                  "duration": "20.807ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "e54915bfcbec7e31:e11994f00000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "3m12s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "invalidate metadata functional_parquet. alltypesinsert",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:39:42.739010000",
                  "end_time": "2018-06-13 12:39:42.757822000",
                  "duration": "18.812ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "c0407d727f51dabb:2d52e66e00000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "3m12s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "invalidate metadata functional_parquet. alltypesinsert",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:39:36.583264000",
                  "end_time": "2018-06-13 12:39:36.603665000",
                  "duration": "20.401ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "1f479ff7bbd40013:6a56043600000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "3m18s",
                  "resource_pool": ""
              },
              {
                  "effective_user": "jenkins",
                  "default_db": "functional_parquet",
                  "stmt": "invalidate metadata functional_parquet. alltypesinsert",
                  "stmt_type": "DDL",
                  "start_time": "2018-06-13 12:39:36.463358000",
                  "end_time": "2018-06-13 12:39:36.487449000",
                  "duration": "24.091ms",
                  "progress": "N/A",
                  "state": "FINISHED",
                  "rows_fetched": 0,
                  "query_id": "164014adb49e2725:8f7fbba900000000",
                  "last_event": "Unregister query",
                  "waiting": true,
                  "executing": false,
                  "waiting_time": "3m19s",
                  "resource_pool": ""
              }
          ],
          "completed_log_size": 25,
          "query_locations": [
              {
                  "location": "ec2-m2-4xlarge-centos-6-4-19ae.vpc.cloudera.com:22002",
                  "count": 0
              },
              {
                  "location": "ec2-m2-4xlarge-centos-6-4-19ae.vpc.cloudera.com:22001",
                  "count": 0
              },
              {
                  "location": "ec2-m2-4xlarge-centos-6-4-19ae.vpc.cloudera.com:22000",
                  "count": 0
              }
          ]
      }"
      13:31:54 E   threadz: "{
          "__common__": {
              "process-name": "impalad",
              "navbar": [
                  {
                      "link": "/",
                      "title": "/"
                  },
                  {
                      "link": "/backends",
                      "title": "/backends"
                  },
                  {
                      "link": "/catalog",
                      "title": "/catalog"
                  },
                  {
                      "link": "/hadoop-varz",
                      "title": "/hadoop-varz"
                  },
                  {
                      "link": "/log_level",
                      "title": "/log_level"
                  },
                  {
                      "link": "/logs",
                      "title": "/logs"
                  },
                  {
                      "link": "/memz",
                      "title": "/memz"
                  },
                  {
                      "link": "/metrics",
                      "title": "/metrics"
                  },
                  {
                      "link": "/queries",
                      "title": "/queries"
                  },
                  {
                      "link": "/rpcz",
                      "title": "/rpcz"
                  },
                  {
                      "link": "/sessions",
                      "title": "/sessions"
                  },
                  {
                      "link": "/threadz",
                      "title": "/threadz"
                  },
                  {
                      "link": "/varz",
                      "title": "/varz"
                  }
              ]
          },
          "total_threads": 30,
          "thread-groups": [
              {
                  "name": "common",
                  "size": 3,
                  "num_created": 3
              },
              {
                  "name": "data-stream-mgr",
                  "size": 16,
                  "num_created": 16
              },
              {
                  "name": "disk-io-mgr",
                  "size": 64,
                  "num_created": 64
              },
              {
                  "name": "disk-io-mgr-handle-cache",
                  "size": 1,
                  "num_created": 1
              },
              {
                  "name": "exec-rpc-pool",
                  "size": 12,
                  "num_created": 12
              },
              {
                  "name": "fragment-execution",
                  "size": 1,
                  "num_created": 11004
              },
              {
                  "name": "hdfs-worker-pool",
                  "size": 16,
                  "num_created": 16
              },
              {
                  "name": "impala-server",
                  "size": 8,
                  "num_created": 8
              },
              {
                  "name": "krpc-data-stream-mgr",
                  "size": 1,
                  "num_created": 1
              },
              {
                  "name": "query-exec-mgr",
                  "size": 0,
                  "num_created": 1866
              },
              {
                  "name": "query-exec-state",
                  "size": 0,
                  "num_created": 4081
              },
              {
                  "name": "rpc-pool",
                  "size": 8,
                  "num_created": 8
              },
              {
                  "name": "scheduling",
                  "size": 1,
                  "num_created": 1
              },
              {
                  "name": "service pool",
                  "size": 8,
                  "num_created": 8
              },
              {
                  "name": "setup-server",
                  "size": 4,
                  "num_created": 4
              },
              {
                  "name": "statestore-subscriber",
                  "size": 1,
                  "num_created": 1
              },
              {
                  "name": "thrift-server",
                  "size": 16,
                  "num_created": 61
              }
          ],
          "jvm-threads": {
              "name": "jvm",
              "total": 76,
              "daemon": 22
          }
      }"
      13:31:54 E   rpcz: "{
          "__common__": {
              "process-name": "impalad",
              "navbar": [
                  {
                      "link": "/",
                      "title": "/"
                  },
                  {
                      "link": "/backends",
                      "title": "/backends"
                  },
                  {
                      "link": "/catalog",
                      "title": "/catalog"
                  },
                  {
                      "link": "/hadoop-varz",
                      "title": "/hadoop-varz"
                  },
                  {
                      "link": "/log_level",
                      "title": "/log_level"
                  },
                  {
                      "link": "/logs",
                      "title": "/logs"
                  },
                  {
                      "link": "/memz",
                      "title": "/memz"
                  },
                  {
                      "link": "/metrics",
                      "title": "/metrics"
                  },
                  {
                      "link": "/queries",
                      "title": "/queries"
                  },
                  {
                      "link": "/rpcz",
                      "title": "/rpcz"
                  },
                  {
                      "link": "/sessions",
                      "title": "/sessions"
                  },
                  {
                      "link": "/threadz",
                      "title": "/threadz"
                  },
                  {
                      "link": "/varz",
                      "title": "/varz"
                  }
              ]
          },
          "servers": [
              {
                  "name": "statestore-subscriber",
                  "methods": [
                      {
                          "name": "StatestoreSubscriber.Heartbeat",
                          "summary": "Count: 10924, min / max: 0 / 1ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                          "in_flight": 0,
                          "server_name": "statestore-subscriber"
                      },
                      {
                          "name": "StatestoreSubscriber.UpdateState",
                          "summary": "Count: 114071, min / max: 0 / 687ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 7ms",
                          "in_flight": 0,
                          "server_name": "statestore-subscriber"
                      }
                  ]
              },
              {
                  "name": "backend",
                  "methods": [
                      {
                          "name": "ImpalaInternalService.PublishFilter",
                          "summary": "Count: 13, min / max: 4ms / 6ms, 25th %-ile: 5ms, 50th %-ile: 6ms, 75th %-ile: 6ms, 90th %-ile: 6ms, 95th %-ile: 6ms, 99.9th %-ile: 6ms",
                          "in_flight": 0,
                          "server_name": "backend"
                      },
                      {
                          "name": "ImpalaInternalService.UpdateFilter",
                          "summary": "Count: 27, min / max: 3ms / 24ms, 25th %-ile: 4ms, 50th %-ile: 4ms, 75th %-ile: 21ms, 90th %-ile: 23ms, 95th %-ile: 23ms, 99.9th %-ile: 24ms",
                          "in_flight": 0,
                          "server_name": "backend"
                      },
                      {
                          "name": "ImpalaInternalService.CancelQueryFInstances",
                          "summary": "Count: 291, min / max: 0 / 46ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 5ms, 99.9th %-ile: 46ms",
                          "in_flight": 0,
                          "server_name": "backend"
                      },
                      {
                          "name": "ImpalaInternalService.ReportExecStatus",
                          "summary": "Count: 8046, min / max: 0 / 51ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 2ms, 99.9th %-ile: 37ms",
                          "in_flight": 0,
                          "server_name": "backend"
                      },
                      {
                          "name": "ImpalaInternalService.ExecQueryFInstances",
                          "summary": "Count: 1866, min / max: 0 / 25ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 1ms, 90th %-ile: 2ms, 95th %-ile: 2ms, 99.9th %-ile: 21ms",
                          "in_flight": 0,
                          "server_name": "backend"
                      }
                  ]
              },
              {
                  "name": "beeswax",
                  "methods": [
                      {
                          "name": "BeeswaxService.get_default_configuration",
                          "summary": "Count: 1, min / max: 2ms / 2ms, 25th %-ile: 2ms, 50th %-ile: 2ms, 75th %-ile: 2ms, 90th %-ile: 2ms, 95th %-ile: 2ms, 99.9th %-ile: 2ms",
                          "in_flight": 0,
                          "server_name": "beeswax"
                      },
                      {
                          "name": "ImpalaService.CloseInsert",
                          "summary": "Count: 316, min / max: 3ms / 63ms, 25th %-ile: 5ms, 50th %-ile: 10ms, 75th %-ile: 16ms, 90th %-ile: 20ms, 95th %-ile: 23ms, 99.9th %-ile: 63ms",
                          "in_flight": 0,
                          "server_name": "beeswax"
                      },
                      {
                          "name": "BeeswaxService.fetch",
                          "summary": "Count: 3086, min / max: 0 / 25s223ms, 25th %-ile: 0, 50th %-ile: 1ms, 75th %-ile: 6ms, 90th %-ile: 24ms, 95th %-ile: 37ms, 99.9th %-ile: 462ms",
                          "in_flight": 0,
                          "server_name": "beeswax"
                      },
                      {
                          "name": "BeeswaxService.close",
                          "summary": "Count: 3653, min / max: 0 / 85ms, 25th %-ile: 1ms, 50th %-ile: 1ms, 75th %-ile: 13ms, 90th %-ile: 21ms, 95th %-ile: 27ms, 99.9th %-ile: 68ms",
                          "in_flight": 0,
                          "server_name": "beeswax"
                      },
                      {
                          "name": "BeeswaxService.get_results_metadata",
                          "summary": "Count: 2209, min / max: 0 / 3ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                          "in_flight": 0,
                          "server_name": "beeswax"
                      },
                      {
                          "name": "ImpalaService.GetRuntimeProfile",
                          "summary": "Count: 3708, min / max: 0 / 1s899ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 3ms, 90th %-ile: 5ms, 95th %-ile: 7ms, 99.9th %-ile: 14ms",
                          "in_flight": 0,
                          "server_name": "beeswax"
                      },
                      {
                          "name": "ImpalaService.Cancel",
                          "summary": "Count: 74, min / max: 1ms / 6ms, 25th %-ile: 2ms, 50th %-ile: 3ms, 75th %-ile: 4ms, 90th %-ile: 5ms, 95th %-ile: 5ms, 99.9th %-ile: 6ms",
                          "in_flight": 0,
                          "server_name": "beeswax"
                      },
                      {
                          "name": "BeeswaxService.get_log",
                          "summary": "Count: 3891, min / max: 0 / 2ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                          "in_flight": 0,
                          "server_name": "beeswax"
                      },
                      {
                          "name": "ImpalaService.GetExecSummary",
                          "summary": "Count: 3704, min / max: 0 / 2ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 2ms",
                          "in_flight": 0,
                          "server_name": "beeswax"
                      },
                      {
                          "name": "BeeswaxService.get_state",
                          "summary": "Count: 23316, min / max: 0 / 2ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                          "in_flight": 0,
                          "server_name": "beeswax"
                      },
                      {
                          "name": "BeeswaxService.query",
                          "summary": "Count: 4097, min / max: 5ms / 29s267ms, 25th %-ile: 9ms, 50th %-ile: 18ms, 75th %-ile: 34ms, 90th %-ile: 3s768ms, 95th %-ile: 5s324ms, 99.9th %-ile: 10s248ms",
                          "in_flight": 0,
                          "server_name": "beeswax"
                      }
                  ]
              },
              {
                  "name": "hs2",
                  "methods": [
                      {
                          "name": "TCLIService.GetTableTypes",
                          "summary": "Count: 1, min / max: 62ms / 62ms, 25th %-ile: 62ms, 50th %-ile: 62ms, 75th %-ile: 62ms, 90th %-ile: 62ms, 95th %-ile: 62ms, 99.9th %-ile: 62ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.GetSchemas",
                          "summary": "Count: 1, min / max: 75ms / 75ms, 25th %-ile: 75ms, 50th %-ile: 75ms, 75th %-ile: 75ms, 90th %-ile: 75ms, 95th %-ile: 75ms, 99.9th %-ile: 75ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.GetFunctions",
                          "summary": "Count: 2, min / max: 5ms / 22ms, 25th %-ile: 5ms, 50th %-ile: 5ms, 75th %-ile: 22ms, 90th %-ile: 22ms, 95th %-ile: 22ms, 99.9th %-ile: 22ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.FetchResults",
                          "summary": "Count: 114, min / max: 0 / 34ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 1ms, 90th %-ile: 4ms, 95th %-ile: 6ms, 99.9th %-ile: 34ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.GetColumns",
                          "summary": "Count: 15, min / max: 5ms / 4s001ms, 25th %-ile: 5ms, 50th %-ile: 6ms, 75th %-ile: 3s406ms, 90th %-ile: 3s924ms, 95th %-ile: 3s924ms, 99.9th %-ile: 4s000ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.CloseSession",
                          "summary": "Count: 9, min / max: 0 / 5ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 1ms, 90th %-ile: 3ms, 95th %-ile: 5ms, 99.9th %-ile: 5ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.CloseOperation",
                          "summary": "Count: 50, min / max: 0 / 14ms, 25th %-ile: 1ms, 50th %-ile: 2ms, 75th %-ile: 4ms, 90th %-ile: 11ms, 95th %-ile: 12ms, 99.9th %-ile: 14ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.GetResultSetMetadata",
                          "summary": "Count: 49, min / max: 0 / 1ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.GetCatalogs",
                          "summary": "Count: 1, min / max: 19ms / 19ms, 25th %-ile: 19ms, 50th %-ile: 19ms, 75th %-ile: 19ms, 90th %-ile: 19ms, 95th %-ile: 19ms, 99.9th %-ile: 19ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.GetOperationStatus",
                          "summary": "Count: 607, min / max: 0 / 1ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 0, 90th %-ile: 0, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.ExecuteStatement",
                          "summary": "Count: 48, min / max: 0 / 666ms, 25th %-ile: 12ms, 50th %-ile: 14ms, 75th %-ile: 21ms, 90th %-ile: 69ms, 95th %-ile: 288ms, 99.9th %-ile: 666ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.GetTables",
                          "summary": "Count: 10, min / max: 4ms / 22ms, 25th %-ile: 5ms, 50th %-ile: 7ms, 75th %-ile: 18ms, 90th %-ile: 18ms, 95th %-ile: 22ms, 99.9th %-ile: 22ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      },
                      {
                          "name": "TCLIService.OpenSession",
                          "summary": "Count: 17, min / max: 0 / 1ms, 25th %-ile: 0, 50th %-ile: 0, 75th %-ile: 1ms, 90th %-ile: 1ms, 95th %-ile: 1ms, 99.9th %-ile: 1ms",
                          "in_flight": 0,
                          "server_name": "hs2"
                      }
                  ]
              }
          ],
          "rpc_connections_accepted": 3,
          "num_inbound_calls_in_flight": 0,
          "per_conn_metrics": [
              {
                  "remote_ip": "127.0.0.1:27002",
                  "outbound_queue_size": 0
              },
              {
                  "remote_ip": "127.0.0.1:27000",
                  "outbound_queue_size": 0
              },
              {
                  "remote_ip": "127.0.0.1:27001",
                  "outbound_queue_size": 0
              }
          ],
          "num_outbound_calls_in_flight": 0,
          "services": [
              {
                  "service_name": "impala.DataStreamService",
                  "queue_size": 0,
                  "idle_threads": 8,
                  "rpcs_queue_overflow": 0,
                  "mem_usage": "0",
                  "mem_peak": "77.29 KB",
                  "incoming_queue_time": "Count: 19314, min / max: 24.000us / 6.016ms, 25th %-ile: 77.000us, 50th %-ile: 92.000us, 75th %-ile: 118.000us, 90th %-ile: 168.000us, 95th %-ile: 231.000us, 99.9th %-ile: 1.629ms",
                  "rpc_method_metrics": [
                      {
                          "method_name": "EndDataStream",
                          "handler_latency": "Count: 5446, min / max: 30.000us / 1.316ms, 25th %-ile: 39.000us, 50th %-ile: 42.000us, 75th %-ile: 54.000us, 90th %-ile: 74.000us, 95th %-ile: 81.000us, 99.9th %-ile: 228.000us",
                          "payload_size": "Count: 5446, min / max: 77.00 B / 80.00 B, 25th %-ile: 78.00 B, 50th %-ile: 79.00 B, 75th %-ile: 79.00 B, 90th %-ile: 80.00 B, 95th %-ile: 80.00 B, 99.9th %-ile: 80.00 B"
                      },
                      {
                          "method_name": "TransmitData",
                          "handler_latency": "Count: 13868, min / max: 88.000us / 2m9s, 25th %-ile: 136.000us, 50th %-ile: 162.000us, 75th %-ile: 211.000us, 90th %-ile: 268.000us, 95th %-ile: 322.000us, 99.9th %-ile: 59s768ms",
                          "payload_size": "Count: 13868, min / max: 101.00 B / 51.17 KB, 25th %-ile: 124.00 B, 50th %-ile: 189.00 B, 75th %-ile: 4.83 KB, 90th %-ile: 11.50 KB, 95th %-ile: 11.91 KB, 99.9th %-ile: 50.91 KB"
                      }
                  ]
              }
          ]
      }"
      

        Attachments

          Activity

            People

            • Assignee:
              kwho Michael Ho
              Reporter:
              tianyiwang Tianyi Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: