Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
The summary is a bit of long. But the basic idea is to better utilize multiple file system partitions.
For example, in a map reduce job, if we have 100 splits local to a node, and these 100 splits spread
across 4 file system partitions, if we allow 4 mappers running concurrently, it is better that mappers
each work on splits on different file system partitions. If in the worst case,
all the mappers work on the splits on the same file system partition, then the other three
file systems are not utilized at all.