[MAPREDUCE-225] Fault tolerant Hadoop Job Tracker

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Incomplete

Environment

High availability enterprise system

• Assignee: Francesco Salbaroli
• Reporter: Francesco Salbaroli
• Created: 04/Nov/08 11:23
• Updated: 18/Jul/14 22:27
• Resolved: 18/Jul/14 22:27

Description

The Hadoop framework has been designed, in an eort to enhance perfor-
mances, with a single JobTracker (master node). It's responsibilities varies
from managing job submission process, compute the input splits, schedule
the tasks to the slave nodes (TaskTrackers) and monitor their health.
In some environments, like the IBM and Google's Internet-scale com-
puting initiative, there is the need for high-availability, and performances
becomes a secondary issue. In this environments, having a system with
a Single Point of Failure (such as Hadoop's single JobTracker) is a major
concern.
My proposal is to provide a redundant version of Hadoop by adding
support for multiple replicated JobTrackers. This design can be approached
in many dierent ways.

In the document at: http://sites.google.com/site/hadoopthesis/Home/FaultTolerantHadoop.pdf?attredirects=0

I wrote an overview of the problem and some approaches to solve it.

I post this to the community to gather feedback on the best way to proceed in my work.

Thank you!

Issue Links

• depends upon: HADOOP-3245 Provide ability to persist running jobs (extend HADOOP-1876)
• depends upon: HADOOP-1876 Persisting completed jobs status
• incorporates: MAPREDUCE-65 TaskTrackers never (re)connect back to the JobTracker if the JobTracker node/machine is changed
• is duplicated by: YARN-149 [Umbrella] ResourceManager (RM) Fail-over
• is related to: MAPREDUCE-2288 JT Availability