[HIVE-12411] Remove counter based stats collection mechanism

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.2.1
• Fix Version/s: 2.0.0
• Component/s: Statistics
• Labels: TODOC2.0
• Assignee: Pengcheng Xiong
• Reporter: Pengcheng Xiong
• Created: 13/Nov/15 21:36
• Updated: 16/Feb/16 23:53
• Resolved: 24/Nov/15 09:27

Description

Following HIVE-12005, HIVE-12164, we have removed jdbc and hbase stats collection mechanism. Now we are targeting counter based stats collection mechanism. The main advantages are as follows (1) counter based stats has limitation on the length of the counter itself, if it is too long, MD5 will be applied. (2) when there are a large number of partitions and columns, we need to create a large number of counters in memory. This will put a heavy load on the M/R AM or Tez AM etc. FS based stats will do a better job.

Issue Links

• relates to: HIVE-12515 Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]
• links to: RB