[IMPALA-9832] Runaway disk usage for ASAN test run

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Not Applicable
• Component/s: Backend
• Labels: broken-build flaky
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-1
• Assignee: Unassigned
• Reporter: Joe McDonnell
• Created: 05/Jun/20 17:35
• Updated: 17/Nov/20 20:30
• Resolved: 17/Nov/20 20:30

Description

A recent ASAN run saw hundreds of test failures due to the HDFS NameNode going into safe mode due to lack of disk space:

2020-06-05 01:03:36,366 WARN org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker: Space available on volume '/dev/nvme0n1p1' is 4419584, which is below the configured reserved amount 104857600
2020-06-05 01:03:36,366 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: NameNode low on available disk space. Entering safe mode.

Metrics about diskspace usage at the time show the device (which had a capacity of 300GB) getting to 99-100% disk usage. Previous runs on the same configuration usually stayed in the 35-40% range. The core job with the same commits also stayed in the 35-40% range.