[HIVE-7438] Counters, statistics, and metrics [Spark Branch]

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: Spark
• Labels: Spark-M3 TODOC-SPARK
• Assignee: Chengxiang Li
• Reporter: Xuefu Zhang
• Created: 17/Jul/14 20:29
• Updated: 05/Dec/14 14:39
• Resolved: 05/Dec/14 02:19

Description

Hive makes use of MapReduce counters for statistics and possibly for other purposes. For Hive on Spark, we should achieve the same functionality using Spark's accumulators.

Hive also collects metrics from MapReduce jobs traditionally. Spark job very likely publishes a different set of metrics, which, if made available, would help user to get insights into their spark jobs. Thus, we should obtain the metrics and make them available as we do for MapReduce.

This task therefore includes:

1. identify Hive's existing functionality w.r.t. counters, statistics, and metrics;
2. design and implement the same functionality in Spark.

Please refer to the design document for more information. https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark#HiveonSpark-CountersandMetrics

Issue Links

• depends upon: HIVE-7706 ClassCastException trying to CTAS table
• depends upon: HIVE-7551 expand spark accumulator to support hive counter [Spark Branch]
• depends upon: HIVE-7552 Collect spark job statistic through spark metrics [Spark Branch]
• incorporates: HIVE-7893 Find a way to get a job identifier when submitting a spark job [Spark Branch]
• is depended upon by: HIVE-7772 Add tests for order/sort/distribute/cluster by query [Spark Branch]