[HIVE-12010] Tests should use FileSystem based stats collection mechanism

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Statistics
• Assignee: Ashutosh Chauhan
• Reporter: Ashutosh Chauhan
• Created: 01/Oct/15 20:14
• Updated: 16/Feb/16 23:52
• Resolved: 13/Oct/15 18:36

Description

Although fs based collection mechanism is default for last few releases, tests still use jdbc for stats collection. The main advantage of fs based collection over jdbc based one is the scalability. In jdbc case, a single database (normally co-located with the metastore relational database) is used to handle all the stats collected by all the tasks. This single database is responsible to maintain the consistency for the stats, which will become a bottleneck and face scalability issue when the number of tasks is huge. In fs case, each task is writing stats into hdfs which does not have scalability issue.

Issue Links

• links to: RB request