[IMPALA-8376] Add per-directory limits for scratch disk usage

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 3.3.0
• Component/s: Backend
• Labels: resource-management
• Epic Color: ghx-label-4
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 29/Mar/19 21:13
• Updated: 10/Aug/19 00:10
• Resolved: 07/Aug/19 03:27

Description

The current syntax is:

--scratch_dirs=/data/1/impala/impalad,/data/10/impala/impalad,/data/11/impala/impalad,/data/2/impala/impalad,/data/3/impala/impalad,/data/4/impala/impalad,/data/5/impala/impalad,/data/6/impala/impalad,/data/7/impala/impalad,/data/8/impala/impalad,/data/9/impala/impalad,/data/12/impala/impalad

The current syntax for the data cache is

--data_cache_dir=/tmp --data_cache_size=500MB

One idea is to allow optionally specifying the limit after each directory:

--scratch_dirs=/data/1/impala/impalad:500MB,/data/10/impala/impalad:2GB,/data/11/impala/impalad

Issue Links

• breaks: IMPALA-8850 TmpFileMgrTest.TestDirectoryLimitParsing failed in asf-master build with error "Value of: dirs3.size() Actual: 3 Expected: 4"