Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4588

Map local segments as on-disk segments

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: task
    • Labels:
      None
    • Target Version/s:

      Description

      Local map segments should never be handled as though they were remote (i.e., copied through a servlet to local disk). This optimization is uniformally more efficient for the fetch, though it increases the number of on-disk segments. Each segment has its own overhead, which can exceed the cost of pulling it into memory (e.g., the decompressor overhead for an active segment exceeds the cost of decompression into memory). Some logic is required to handle a large number of such segments.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              cdouglas Christopher Douglas

              Dates

              • Created:
                Updated:

                Issue deployment