[MAPREDUCE-4588] Map local segments as on-disk segments

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: task
• Assignee: Unassigned
• Reporter: Christopher Douglas
• Created: 25/Aug/12 00:17
• Updated: 25/Aug/12 00:20

Description

Local map segments should never be handled as though they were remote (i.e., copied through a servlet to local disk). This optimization is uniformally more efficient for the fetch, though it increases the number of on-disk segments. Each segment has its own overhead, which can exceed the cost of pulling it into memory (e.g., the decompressor overhead for an active segment exceeds the cost of decompression into memory). Some logic is required to handle a large number of such segments.

Issue Links

• is part of: MAPREDUCE-4584 Umbrella: Preemption and restart of MapReduce tasks