Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-5494

IFile.Reader should have a nextRawKey/nextRawValue

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.18.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Merger.Segment has only the next() method defined which internally calls next(key,value) on the underlying IFile stream. This would read both the key and the value bytes. It would be good to have Merger.Segment.nextRawKey(), that would read only the key and delay reading the value until needed (in Merger.MergeQueue.next()) via a new method Merger.Segment.nextRawValue().
      This would mean that we load only one value bytes at a time, and hence would incur potentially much less (depending on how big the values are) on the memory footprint.

        Attachments

        1. 5494-1.patch
          13 kB
          Devaraj Das
        2. 5494-2.patch
          14 kB
          Devaraj Das
        3. 5494-3.patch
          15 kB
          Devaraj Das
        4. 5494-4.patch
          15 kB
          Devaraj Das

          Issue Links

            Activity

              People

              • Assignee:
                devaraj Devaraj Das
                Reporter:
                devaraj Devaraj Das
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: