Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4866

ShuffleRamManager is limited to 2Gb of memory - we should increase that

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 0.20.2
    • Fix Version/s: None
    • Component/s: mrv1
    • Labels:
    • Environment:

      linux, 64bits cpu, more than 2Gb of memory for each reducer tasks

      Description

      Inside the org.apache.hadoop.mapred.ReduceTask.java, the ShuffleRamManager is limited to allocate up to 2Gb of memory during the shuffle phase.
      We should be able to allocate more, to take advantage of the full memory we have on servers.

      1. MAPREDUCE-4866-INCOMPLETE.patch
        4 kB
        Varene Olivier
      2. M4866-0.patch
        1 kB
        Chris Douglas

        Activity

        Varene Olivier created issue -
        Hide
        Varene Olivier added a comment -

        This patch is incomplete and does not work !!!
        because the allocation of byte array is limited to around Integer.MAX_VALUE,

        we need to find another way to allocate such huge space of memory.

        What about BigArrays ?
        What do you think ? propose ?

        Show
        Varene Olivier added a comment - This patch is incomplete and does not work !!! because the allocation of byte array is limited to around Integer.MAX_VALUE, we need to find another way to allocate such huge space of memory. What about BigArrays ? What do you think ? propose ?
        Varene Olivier made changes -
        Field Original Value New Value
        Attachment MAPREDUCE-4866-INCOMPLETE.patch [ 12560164 ]
        Chris Douglas made changes -
        Attachment M4866-0.patch [ 12560381 ]
        Hide
        Varene Olivier added a comment -

        Sorry Chris,

        but I think your patch is incomplete, only changing maxSize is not enough,

        please have a look to the proposed patch MAPREDUCE-4866-INCOMPLETE.patch with all the changes proposed
        the patch is incomplete because of array size limitation to around Integer.MAX_VALUE

        ReduceTask.java > Class MapOutputCopier > method ShuffleInMemory
        line 1507: byte[] shuffleData = new byte[mapOutputLength];

        we need to change this allocation with some kind of big array (whose size can be a long)

        regards
        Olivier

        Show
        Varene Olivier added a comment - Sorry Chris, but I think your patch is incomplete, only changing maxSize is not enough, please have a look to the proposed patch MAPREDUCE-4866 -INCOMPLETE.patch with all the changes proposed the patch is incomplete because of array size limitation to around Integer.MAX_VALUE ReduceTask.java > Class MapOutputCopier > method ShuffleInMemory line 1507: byte[] shuffleData = new byte [mapOutputLength] ; we need to change this allocation with some kind of big array (whose size can be a long) regards Olivier
        Hide
        Chris Douglas added a comment -

        The patch is against the 1.x line; the 0.20 line is not actively developed. The changes to make these longs were in MAPREDUCE-1182.

        Pulling a >2GB segment into memory is... optimistic. It's a very specialized job that would benefit from relaxing this constraint.

        Show
        Chris Douglas added a comment - The patch is against the 1.x line; the 0.20 line is not actively developed. The changes to make these longs were in MAPREDUCE-1182 . Pulling a >2GB segment into memory is... optimistic. It's a very specialized job that would benefit from relaxing this constraint.
        Harsh J made changes -
        Link This issue is duplicated by MAPREDUCE-5705 [ MAPREDUCE-5705 ]
        Harsh J made changes -
        Link This issue is duplicated by MAPREDUCE-5705 [ MAPREDUCE-5705 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Varene Olivier
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development