[HIVE-20025] Clean-up of event files created by HiveProtoLoggingHook.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.2.0, 4.0.0-alpha-1
• Component/s: HiveServer2
• Labels: Hive hooks pull-request-available
• Target Version/s: 3.2.0, 4.0.0
• Assignee: Sankar Hariappan
• Reporter: Sankar Hariappan
• Created: 28/Jun/18 17:08
• Updated: 17/Nov/22 08:54
• Resolved: 05/Jul/18 11:50

Description

Currently, HiveProtoLoggingHook write event data to hdfs. The number of files can grow to very large numbers.

Since the files are created under a folder with Date being a part of the path, hive should have a way to clean up data older than a certain configured time / date. This can be a job that can run with as little frequency as just once a day.

This time should be set to 1 week default. There should also be a sane upper bound of # of files so that when a large cluster generates a lot of files during a spike, we don't force the cluster fall over.

Issue Links

• links to: GitHub Pull Request #384