[HIVE-2446] Introduction of client statistics publishers possibility

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: Clients, Statistics
• Release Note: See https://reviews.apache.org/r/2182/diff/#index_header
• Assignee: Robert Surówka
• Reporter: Robert Surówka
• Created: 14/Sep/11 21:23
• Updated: 21/Mar/14 06:04
• Resolved: 05/Oct/11 21:36

Description

The purpose of this change is to allow publication or storage of counters while the job is running.

Introduced two new variables to hive-default.xml and HiveConf.java: "hive.client.stats.publishers" and "hive.client.stats.counters". First one specifies classes names, whose instances will be executed by HadoopJobExecHelper.java (similarly as hooks are) in its method progress(ExecDriverTaskHandle): MapRedStats. Second one specifies list of counters that any client stat publishers should publish or stored. Details regarding format of this list is up to a specific deployment (it is up to client stats publishers to parse it), yet it is required to use display names of counter groups and counters.

Added interface ClientStatsPublishers in org.apache.hadoop.hive.ql.stats package, that must be implemented by all stats publishers.

Added code to progress(ExecDriverTaskHandle): MapRedStats from HadoopJobExecHelper.java that puts counters' values to a Java map and then executes registered client stats publishers giving them that map and running job id. Added two new methods to HadoopJobExecHelper: extractAllCounterValues(Counters) and getClientStatsPublishers() that are used by code from previous sentence.

Made cosmetic changes in two other classes

Issue Links

• is related to: HIVE-2522 HIVE-2446 bug (next one) - If constructor of ClientStatsPublisher throws runtime exception it will be propagated to HadoopJobExecHelper's progress method and beyond, whereas it shouldn't
• is related to: HIVE-2487 Bug from HIVE-2446, the code that calls client stats publishers run() methods is in wrong place, should be in the same method but inside of while (!rj.isComplete()) {} loop