[IMPALA-8374] Add high water mark metrics and limits for spill disk usage

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.2.0
• Fix Version/s: Impala 3.3.0
• Component/s: Backend
• Labels: observability resource-management
• Epic Color: ghx-label-6
• Assignee: Unassigned
• Reporter: David Rorke
• Created: 29/Mar/19 19:02
• Updated: 22/Aug/19 21:56
• Resolved: 22/Aug/19 21:56

Description

There is currently no way to limit the disk usage for spills on a per host/volume basis or monitor the peak disk usage (high water mark) for spilling.   Both of these features would make it easier for users to manage the disk resources used for spilling when the spill usage is competing with other uses of the same volume.

Sub-Tasks

• Add metrics for spill disk usage: Closed
• Add per-directory limits for scratch disk usage: Resolved
• Document IMPALA-8376 Add per-directory limits for scratch disk usage: Closed