Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
None
-
None
-
None
-
None
Description
Job submission involves two steps: submitting jobs to the System directory on DFS (done by the client), then submit the job via the JobSubmissionProtocol to JobTracker. This two step process is seen to have some issues:
- Since the files need to be read from DFS, slowness in the DFS can cause job initialization to become costly. We faced this as described in
HADOOP-5286andHADOOP-4664. - The two step process could lead to inconsistent information being left around - like in
HADOOP-5327andHADOOP-5335.
This JIRA is to explore options to remove the two step process in submitting a job.