Another approach to this would be to have an admin command for the job tracker that would have the job tracker stop accepting new jobs and shut itself down when the last job completed. That way the external process wouldn't need to continually poll the job tracker to see if it was done yet. Thoughts?
The requirement we have in Hadoop On Demand is to shutdown a job tracker when it has not run any jobs for a period of time. This is to help in releasing unused resources, and reclaim them. The approach suggested above actually makes sense if we want to run the job tracker for a period of time and then shutdown. This seems to be different from what we need, right ?
The problem of continous polling does exist. But I feel it may not be too intensive. Any alternative that does not have this problem, but solves the use-case would also be great.