[SPARK-2508] fail to remove staging dir when app quits in a yarn-cluster

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 0.9.0, 0.9.1, 1.0.0
• Component/s: Spark Core

Environment

RedHat 5.3
2.0.0-cdh4.6.0
java version "1.6.0_45"

• Target Version/s: 0.9.0, 0.9.1, 1.0.0
• Assignee: Unassigned
• Reporter: James Z.M. Gao
• Created: 16/Jul/14 03:36
• Updated: 26/Feb/15 01:04
• Resolved: 26/Feb/15 01:04

Description

Hi,

In a yarn-cluster, the driver is actually running as 'yarn' user in some container by a nodemanager. When
submitting jobs from other users, e.g. user1, the stagingDir is created in submitting stage, which is actually a progress run by user1. So the full path of stagingDir on the hdfs should look like '/user/user1/.staging/...'.

And in cleaning up stage, stagingDir could be removed by the driver. At this time, we need give stagingDir a full path, so
the driver could remove the correct directory ('/user/user1/.staging/...'), not '/user/yarn/.staging/...'.

Here is a simple PR: https://github.com/apache/spark/pull/154