[HIVE-14521] codahale metrics exceptions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Vaibhav Gumashta
• Reporter: Sergey Shelukhin
• Created: 11/Aug/16 20:07
• Updated: 12/Aug/16 23:27
• Resolved: 12/Aug/16 23:27

Description

One some random setup, I see bazillions of errors like this in HS2 log:

2016-08-08 04:52:18,619 WARN  [HiveServer2-Handler-Pool: Thread-101]: log.PerfLogger (PerfLogger.java:beginMetrics(226)) - Error recording metrics
java.io.IOException: Scope named api_Driver.run is not closed, cannot be opened.
        at org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics$CodahaleMetricsScope.open(CodahaleMetrics.java:133)
        at org.apache.hadoop.hive.common.metrics.metrics2.CodahaleMetrics.startStoredScope(CodahaleMetrics.java:220)
        at org.apache.hadoop.hive.ql.log.PerfLogger.beginMetrics(PerfLogger.java:223)
        at org.apache.hadoop.hive.ql.log.PerfLogger.PerfLogBegin(PerfLogger.java:143)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:378)
        at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:320)
        at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1214)
        at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1208)
        at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:146)
        at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:226)
        at org.apache.hive.service.cli.operation.Operation.run(Operation.java:276)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:468)
        at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:456)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

I suspect that either, just like the metastore deadline, this needs better error handling when whatever the metrics surround fails; or, it is just not thread safe.
But I actually haven't looked at the code yet.

Issue Links

• duplicates: HIVE-13410 PerfLog metrics scopes not closed if there are exceptions on HS2
• is related to: HIVE-13410 PerfLog metrics scopes not closed if there are exceptions on HS2