[KUDU-2279] Enable metrics logging by default

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.0
• Fix Version/s: 1.7.0
• Component/s: ops-tooling
• Target Version/s: 1.7.0
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 01/Feb/18 03:20
• Updated: 03/May/21 03:44
• Resolved: 06/Feb/18 08:09

Description

In a lot of cases, the granular metrics that we collect can be very useful to understand performance problems. Our internal metrics tend to be much more granular than what is supported by common metrics aggregation systems, so having the local metrics log is useful even in the context of a managed deployment. So, I'd like to move towards enabling metrics logging by default.

Currently, however, the metrics log can be quite verbose, since it's equivalent to dumping all of the /metricsjson endpoint on a periodic basis. Even though we gzip it, we burn a lot of CPU serializing a huge blob (eg /metricsjson is 87MB on a tserver in one of our test clusters). Additionally, we don't currently have the ability to delete old metrics logs automatically, so we'd be in danger of filling up the log dir.

This JIRA tracks various bits of work on the way to enabling metrics logging by default.

Issue Links

• is duplicated by: KUDU-3279 Limit the number of diagnostic files kept