Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.1
    • Component/s: mr-am, mrv2, performance
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      MR AM creates new JobConf objects unnecessarily in a couple of places in JobImpl and TaskImpl which occupy non-trivial amount of heap.

      While working with a 64 bit JVM on 100K maps jobs, with uncompressed pointers, removing those extraneous objects helped in addressing OOM with 2GB AM heap size.

      1. MAPREDUCE-3567-20111222.txt
        26 kB
        Vinod Kumar Vavilapalli
      2. MAPREDUCE-3567-20111215.1.txt
        25 kB
        Vinod Kumar Vavilapalli

        Activity

        Vinod Kumar Vavilapalli created issue -
        Vinod Kumar Vavilapalli made changes -
        Field Original Value New Value
        Component/s mrv2 [ 12314301 ]
        Component/s performance [ 12316500 ]
        Vinod Kumar Vavilapalli made changes -
        Fix Version/s 0.23.1 [ 12318883 ]
        Affects Version/s 0.23.0 [ 12315570 ]
        Description MR AM creates new JobConf objects unnecessarily in a couple of places in JobImpl and TaskImpl which occupy non-trivial amount of heap.

        While working with a 64 bit JVM on 100K maps jobs, with uncompressed pointers, removing those extraneous objects helped in addressing OOM with 2GB AM heap size.
        Component/s mr-am [ 12315342 ]
        Vinod Kumar Vavilapalli made changes -
        Attachment MAPREDUCE-3567-20111215.1.txt [ 12507601 ]
        Vinod Kumar Vavilapalli made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Vinod Kumar Vavilapalli made changes -
        Attachment MAPREDUCE-3567-20111222.txt [ 12508430 ]
        Vinod Kumar Vavilapalli made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hadoop Flags Reviewed [ 10343 ]
        Siddharth Seth made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Target Version/s 0.23.1 [ 12318883 ]
        Resolution Fixed [ 1 ]
        Arun C Murthy made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Vinod Kumar Vavilapalli
            Reporter:
            Vinod Kumar Vavilapalli
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development