Ability to throttle DFS/MR so as not to overwhelm colo to colo switches

• Key: HDFS-355
• Type: Improvement
• Status: Open
• Priority: Major
• Assignee: Unassigned
• Reporter: Pete Wyckoff

Description

Motivation:

This would allow people to put data that is not used as often in non co-located HDFS instance and when needed pulling it from the other cluster.
This is useful in the context of Hive where a Metastore tells the runtime system where the data is located (the full URI) or symbolic links.

The problem:

This will not work right now because it may overwhelm switches between the two instances.

Workaround:

Make the files unplittable or make your block size such that you only get 2-3 mappers.

Possible solution:

Throttle parallelism in the scheduler by specifying to run only X mappers for a job no matter how many slots are free. (making some assumptions about the reliability of the JobTracker's failure detector).

Comments

[Pete Wyckoff]

If we think of the switch as a resource (or at a higher level, the hdfs instance in the other colo as a resource for "this" mapred instance - still not truly global across all mapred instances) this jira relates to HADOOP-3421.