[IMPALA-8537] Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.3.0
• Fix Version/s: Impala 3.3.0
• Component/s: Backend
• Epic Color: ghx-label-3
• Assignee: Abhishek Rawat
• Reporter: David Rorke
• Created: 10/May/19 22:13
• Updated: 04/Jun/19 16:55
• Resolved: 21/May/19 21:07

Description

I'm running a workload that does a lot of spilling and noticed the value reported for tmp-file-mgr.scratch-space-bytes-used is negative on all nodes.

Some details of the workload and cluster configuration:

• Generating a 10 TB TPC-DS partitioned parquet data set (very large sort).
• 30 impalads, each with 48 GB RAM and 14 scratch directories (each on a separate drive)
• Rough estimate (based on query metrics) of total cumulative aggregate memory spilled across the cluster since restart is 6.5 TB.

Snapshot of the bad metrics attached.

Issue Links

• is broken by: IMPALA-8375 Add metrics for spill disk usage