[MAPREDUCE-458] Adding caching to Hadoop which is independent of the task trackers.

• Type: New Feature
• Status: Resolved
• Priority: Minor
• Resolution: Not A Problem
• Assignee: Owen O'Malley
• Reporter: Mahadev Konar
• Created: 12/Aug/06 00:01
• Updated: 16/Jan/12 09:51
• Resolved: 16/Jan/12 09:51

Description

It would be nice to have a feature in Hadoop that could cache files locally that is independent of TaskTrackers and JobTrackers. Hadoop-288 caching is dependent on the tasktrackers. In an environment where you would dynamically bring up and down the TaskTrackers for resource sharing, that is problematic. It would be good to have this feature wherein you can install tasktrackers/jobtrackers on these machines using this caching mechanism. The caching feature could use something like Bittorent /http/rsync to copy the main hadoop.jar.

Issue Links

• relates to: HADOOP-288 RFC: Efficient file caching