[SPARK-6692] Add an option for client to kill AM when it is killed

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 1.3.0
• Component/s: YARN
• Labels: yarn
• Assignee: Cheolsoo Park
• Reporter: Cheolsoo Park
• Created: 03/Apr/15 06:32
• Updated: 01/Jun/15 19:27
• Resolved: 01/Jun/15 19:27

Description

I understand that the yarn-cluster mode is designed for fire-and-forget model; therefore, terminating the yarn client doesn't kill AM.

However, it is very common that users submit Spark jobs via job scheduler (e.g. Apache Oozie) or remote job server (e.g. Netflix Genie) where it is expected that killing the yarn client will terminate AM.

It is true that the yarn-client mode can be used in such cases. But then, the yarn client sometimes needs lots of heap memory for big jobs if it runs in the yarn-client mode. In fact, the yarn-cluster mode is ideal for big jobs because AM can be given arbitrary heap memory unlike the yarn client. So it would be very useful to make it possible to kill AM even in the yarn-cluster mode.

In addition, Spark jobs often become zombie jobs if users ctrl-c them as soon as they're accepted (but not yet running). Although they're eventually shutdown after AM timeout, it would be nice if AM could immediately get killed in such cases too.

Issue Links

• relates to: SPARK-3591 Provide "fire and forget" option for YARN cluster mode
• links to: [Github] Pull Request #5343 (piaozhexiu)