[HIVE-6500] Stats collection via filesystem

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Component/s: Statistics
• Assignee: Ashutosh Chauhan
• Reporter: Ashutosh Chauhan
• Created: 25/Feb/14 07:56
• Updated: 15/Jun/15 12:53
• Resolved: 02/Mar/14 19:05

Description

Recently, support for stats gathering via counter was added Although, its useful it has following issues:

• Length of counter group name is limited
• Length of counter name is limited
• Number of distinct counter groups are limited
• Number of distinct counters are limited
  Although, these limits are configurable, but setting them to higher value implies increased memory load on AM and job history server.
  Now, whether these limits makes sense or not is debatable it is desirable that Hive doesn't make use of counters features of framework so that it we can evolve this feature without relying on support from framework. Filesystem based counter collection is a step in that direction.

Issue Links

• relates to: HIVE-4632 Use hadoop counter as a stat publisher
• links to: RB request