Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: mrv2, performance
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      Removes two sets of previously available config properties:

      1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and,
      2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb )

      In favor of two new, generically named properties:

      1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers.
      2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers.

      Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides.

      Also changes the default minimum and maximums to 128 MB and 10 GB respectively.
      Show
      Removes two sets of previously available config properties: 1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and, 2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb ) In favor of two new, generically named properties: 1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers. 2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers. Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides. Also changes the default minimum and maximums to 128 MB and 10 GB respectively.
    • Target Version/s:

      Description

      After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.

      This is just a configuration change.

      [P.s.]:

      • Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
      • The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
      • 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
      1. MAPREDUCE-3812.patch
        21 kB
        Harsh J
      2. MAPREDUCE-3812.patch
        20 kB
        Harsh J
      3. MAPREDUCE-3812.patch
        21 kB
        Harsh J
      4. MAPREDUCE-3812.patch
        25 kB
        Arun C Murthy
      5. MAPREDUCE-3812.patch
        21 kB
        Arun C Murthy
      6. MAPREDUCE-3812-20120206.1.txt
        19 kB
        Vinod Kumar Vavilapalli
      7. MAPREDUCE-3812-20120206.txt
        10 kB
        Vinod Kumar Vavilapalli
      8. MAPREDUCE-3812-20120205.txt
        10 kB
        Vinod Kumar Vavilapalli

        Issue Links

          Activity

          Vinod Kumar Vavilapalli created issue -
          Vinod Kumar Vavilapalli made changes -
          Field Original Value New Value
          Component/s mrv2 [ 12314301 ]
          Component/s performance [ 12316500 ]
          Vinod Kumar Vavilapalli made changes -
          Assignee Vinod Kumar Vavilapalli [ vinodkv ]
          Fix Version/s 0.23.1 [ 12318883 ]
          Affects Version/s 0.23.0 [ 12315570 ]
          Description After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.

          This is just a configuration change.
          Vinod Kumar Vavilapalli made changes -
          Link This issue is related to MAPREDUCE-3719 [ MAPREDUCE-3719 ]
          Vinod Kumar Vavilapalli made changes -
          Summary Change default AM slot size to be 1GB Change default memory slot sizes to be 1.5GB
          Vinod Kumar Vavilapalli made changes -
          Attachment MAPREDUCE-3812-20120205.txt [ 12513379 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment MAPREDUCE-3812-20120206.txt [ 12513476 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment MAPREDUCE-3812-20120206.1.txt [ 12513504 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-3812.patch [ 12513549 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Arun C Murthy made changes -
          Attachment MAPREDUCE-3812.patch [ 12513566 ]
          Arun C Murthy made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Fix Version/s 0.23.2 [ 12319851 ]
          Fix Version/s 0.23.1 [ 12318883 ]
          Harsh J made changes -
          Fix Version/s 0.23.2 [ 12319851 ]
          Target Version/s 0.23.3 [ 12320060 ]
          Harsh J made changes -
          Link This issue is superceded by MAPREDUCE-4026 [ MAPREDUCE-4026 ]
          Harsh J made changes -
          Assignee Vinod Kumar Vavilapalli [ vinodkv ] Harsh J [ qwertymaniac ]
          Harsh J made changes -
          Target Version/s 0.23.3 [ 12320060 ] 2.0.0, trunk [ 12320354, 12320360 ]
          Harsh J made changes -
          Summary Change default memory slot sizes to be 1.5GB Lower default allocation sizes, fix allocation configurations and document them
          Target Version/s trunk, 2.0.0 [ 12320360, 12320354 ] 2.0.0, trunk [ 12320354, 12320360 ]
          Description After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.

          This is just a configuration change.
          After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.

          This is just a configuration change.

          [P.s.]:
          - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
          - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
          - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
          Harsh J made changes -
          Attachment MAPREDUCE-3812.patch [ 12523646 ]
          Harsh J made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s trunk, 2.0.0 [ 12320360, 12320354 ] 2.0.0, trunk [ 12320354, 12320360 ]
          Harsh J made changes -
          Hadoop Flags Incompatible change [ 10342 ]
          Target Version/s trunk, 2.0.0 [ 12320360, 12320354 ] 2.0.0, trunk [ 12320354, 12320360 ]
          Harsh J made changes -
          Link This issue is duplicated by MAPREDUCE-4026 [ MAPREDUCE-4026 ]
          Harsh J made changes -
          Link This issue is duplicated by MAPREDUCE-4027 [ MAPREDUCE-4027 ]
          Harsh J made changes -
          Attachment MAPREDUCE-3812.patch [ 12523664 ]
          Harsh J made changes -
          Attachment MAPREDUCE-3812.patch [ 12523665 ]
          Robert Joseph Evans made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Target Version/s trunk, 2.0.0 [ 12320360, 12320354 ] 2.0.0, trunk [ 12320354, 12320360 ]
          Fix Version/s 0.23.3 [ 12320060 ]
          Fix Version/s 2.0.0 [ 12320354 ]
          Fix Version/s 3.0.0 [ 12320355 ]
          Resolution Fixed [ 1 ]
          Harsh J made changes -
          Release Note Removed two sets of previously available config properties:

          1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and,
          2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb )

          In favor of two new, generically named properties:

          1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers.
          2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers.

          Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides.

          Also changes the default minimum and maximums to 128 MB and 10 GB respectively.
          Target Version/s trunk, 2.0.0 [ 12320360, 12320354 ] 2.0.0, trunk [ 12320354, 12320360 ]
          Harsh J made changes -
          Release Note Removed two sets of previously available config properties:

          1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and,
          2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb )

          In favor of two new, generically named properties:

          1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers.
          2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers.

          Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides.

          Also changes the default minimum and maximums to 128 MB and 10 GB respectively.
          Removes two sets of previously available config properties:

          1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and,
          2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb )

          In favor of two new, generically named properties:

          1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers.
          2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers.

          Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides.

          Also changes the default minimum and maximums to 128 MB and 10 GB respectively.
          Target Version/s trunk, 2.0.0 [ 12320360, 12320354 ] 2.0.0, trunk [ 12320354, 12320360 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Arun C Murthy made changes -
          Fix Version/s 2.0.2-alpha [ 12322471 ]
          Fix Version/s 2.0.0-alpha [ 12320354 ]
          Fix Version/s 3.0.0 [ 12320355 ]

            People

            • Assignee:
              Harsh J
              Reporter:
              Vinod Kumar Vavilapalli
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development