[MAPREDUCE-6984] MR AM to clean up temporary files from previous attempt in case of no recovery - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.0.0-beta1
Fix Version/s: 3.1.0, 3.0.1, 2.10.0
Component/s: applicationmaster
Labels:
None

Hadoop Flags:

Reviewed

Description

When the MR AM restarts, the {outputDir}/_temporary/{appAttemptNumber} directory remains on HDFS, even though this directory is not used during the next attempt if the restart has been done without recovery. So if recovery is not used for the AM restart, then the deletion of this directory can be done earlier (at the start of the next attempt). The benefit is that more free HDFS space is available for the next attempt.

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

MAPREDUCE-6984.000.patch
17/Oct/17 11:31
18 kB
Gergo Repas
MAPREDUCE-6984.001.patch
17/Oct/17 13:18
18 kB
Gergo Repas
MAPREDUCE-6984.003.patch
04/Jan/18 17:42
19 kB
Gergo Repas
MAPREDUCE-6984.004.patch
05/Jan/18 10:24
19 kB
Gergo Repas
MAPREDUCE-6984.005.patch
05/Jan/18 11:31
20 kB
Gergo Repas
MAPREDUCE-6984.006.patch
08/Jan/18 10:53
20 kB
Gergo Repas
MAPREDUCE-6984.007.patch
19/Jan/18 11:18
25 kB
Gergo Repas
MAPREDUCE-6984.008.patch
19/Jan/18 12:48
24 kB
Gergo Repas
MAPREDUCE-6984.009.patch
19/Jan/18 16:56
9 kB
Gergo Repas
MAPREDUCE-6984.010.patch
19/Jan/18 18:52
8 kB
Gergo Repas

Issue Links

breaks

MAPREDUCE-7041 MR should not try to clean up at first job attempt

Resolved

Activity

People

Assignee:: Gergo Repas

Reporter:: Gergo Repas

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 17/Oct/17 11:14

Updated:: 25/Jan/18 01:22

Resolved:: 19/Jan/18 21:16