Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
LLAP has several R/W locks for serialization of updates to query tracker, file data, ....
Instrumentation is added to monitor the
- total amount of R/W locks within a particular category
- average + max wait/suspension time to get the R/W lock
A category includes all lock instances for particular areas (i.e. category is FileData and all R/W locks that are used in FileData instances are accounted within the one category).
The monitoring/accounting is done via Hadoop Metrics 2, making them accessible via JMX. In addition, a new "locking" GET endpoint is added to the LLAP daemon's REST interface. It produces output like the following example:
{
"statsCollection": "enabled",
"lockStats": [
{ "type": "R/W Lock Stats",
"label": "FileData",
"totalLockWaitTimeMillis": 0,
"readLock": {
"count": 0,
"avgWaitTimeNanos": 0,
"maxWaitTimeNanos": 0
},
"writeLock": {
"count": 0,
"avgWaitTimeNanos": 0,
"maxWaitTimeNanos": 0
}
},
{ "type": "R/W Lock Stats",
"label": "QueryTracker",
"totalLockWaitTimeMillis": 0,
"readLock": {
"count": 0,
"avgWaitTimeNanos": 0,
"maxWaitTimeNanos": 0
},
"writeLock": {
"count": 0,
"avgWaitTimeNanos": 0,
"maxWaitTimeNanos": 0
}
} ]
}
To avoid the overhead of lock instrumentation, lock metrics collection is disabled by default and can be enabled via the following configuration parameter:
hive.llap.lockmetrics.collect = true