[SPARK-19264] Work should start driver, the same to AM of yarn

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Assignee: Unassigned
• Reporter: John Fang
• Created: 17/Jan/17 17:46
• Updated: 30/Jan/17 10:53
• Resolved: 30/Jan/17 10:53

Description

I think work can't start driver by "ProcessBuilderLike", thus we can't know the application's main thread is finished or not if the application's main thread contains some non-daemon threads. Because the program terminates when there no longer is any non-daemon thread running (or someone called System.exit). The main thread can have finished long ago.
worker should start driver like AM of YARN . As followed:

ApplicationMaster.scala

    
     mainMethod.invoke(null, userArgs.toArray)
     finish(FinalApplicationStatus.SUCCEEDED, ApplicationMaster.EXIT_SUCCESS)
     logDebug("Done running users class")

Then the work can monitor the driver's main thread, and know the application's state.