[IMPALA-9435] Usability enhancements for the data cache access trace

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-4
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 27/Feb/20 00:47
• Updated: 17/Jun/20 15:05
• Resolved: 17/Jun/20 15:05

Description

In IMPALA-8542, we added an access trace to the data cache. This outputs a JSON entry with information about each hit/miss. Currently it is controlled by the data_cache_enable_tracing startup parameter, and it will trace all accesses to a single file. There are a few enhancements that would make this easier to enable:

1. Limit the number of access trace entries stored on disk to avoid unlimited disk usage. This can be done by switching to use be/src/util/simple-logger.h rather than a single file. The number of retained entries should be configurable.
2. Implement a way to trace a subset of accesses (e.g. 5%)
3. Optionally provide a way to start/stop logging without restart (e.g. via the WebUI)