Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.0, 2.0.0-alpha
    • Fix Version/s: 0.23.1
    • Component/s: mrv2
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed MR AM recovery so that only single selected task output is recovered and thus reduce the unnecessarily bloated recovery time.

      Description

      Reported by Karam Singh

      yarn.resourcemanager.am.max-retries=2
      Ran test cases with sort job on 350 scale having 16800 maps and 680 reduces -:
      1. After 70 secs of Job Sumbission Am is killed using kill -9, around 3900 maps were completed and 680 reduces were
      scheduled, Second AM got restart. Job got completed in 980 secs. AM took very less time to recover.
      2. After 150 secs of Job Sumbission AM is killed using kill -9, around 90% maps were completed and 680 reduces were
      scheduled , Second AM got restart Job got completed in 1000 secs. AM got revocer.
      3. After 150 secs of Job Sumbission AM as killed using kill -9, almost all maps were completed and only 680 reduces
      were running, Recovery was too slow, AM was still revocering after 1hr :40 mis when I killed the run.

      1. MAPREDUCE-3711-20120203.txt
        83 kB
        Vinod Kumar Vavilapalli
      2. MR-3711.txt
        82 kB
        Robert Joseph Evans
      3. MR-3711.txt
        44 kB
        Robert Joseph Evans
      4. MR-3711.txt
        38 kB
        Robert Joseph Evans

        Issue Links

          Activity

          Siddharth Seth created issue -
          Mahadev konar made changes -
          Field Original Value New Value
          Summary AppMaster resovery for Medium to large jobs take long time AppMaster recovery for Medium to large jobs take long time
          Arun C Murthy made changes -
          Priority Critical [ 2 ] Blocker [ 1 ]
          Robert Joseph Evans made changes -
          Assignee Robert Joseph Evans [ revans2 ]
          Robert Joseph Evans made changes -
          Attachment MR-3711.txt [ 12512794 ]
          Robert Joseph Evans made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s 0.24.0 [ 12317654 ]
          Target Version/s 0.23.1, 0.24.0 [ 12318883, 12317654 ]
          Robert Joseph Evans made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 0.24.0, 0.23.1 [ 12317654, 12318883 ] 0.23.1, 0.24.0 [ 12318883, 12317654 ]
          Robert Joseph Evans made changes -
          Attachment MR-3711.txt [ 12512860 ]
          Robert Joseph Evans made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.24.0, 0.23.1 [ 12317654, 12318883 ] 0.23.1, 0.24.0 [ 12318883, 12317654 ]
          Vinod Kumar Vavilapalli made changes -
          Parent MAPREDUCE-2692 [ 12514289 ]
          Issue Type Bug [ 1 ] Sub-task [ 7 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 0.24.0, 0.23.1 [ 12317654, 12318883 ] 0.23.1, 0.24.0 [ 12318883, 12317654 ]
          Fix Version/s 0.23.1 [ 12318883 ]
          Robert Joseph Evans made changes -
          Attachment MR-3711.txt [ 12513168 ]
          Robert Joseph Evans made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.24.0, 0.23.1 [ 12317654, 12318883 ] 0.23.1, 0.24.0 [ 12318883, 12317654 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 0.24.0, 0.23.1 [ 12317654, 12318883 ] 0.23.1, 0.24.0 [ 12318883, 12317654 ]
          Vinod Kumar Vavilapalli made changes -
          Attachment MAPREDUCE-3711-20120203.txt [ 12513192 ]
          Vinod Kumar Vavilapalli made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hadoop Flags Reviewed [ 10343 ]
          Target Version/s 0.24.0, 0.23.1 [ 12317654, 12318883 ] 0.23.1, 0.24.0 [ 12318883, 12317654 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Release Note Fixed MR AM recovery so that only single selected task output is recovered and thus reduce the unnecessarily bloated recovery time.
          Target Version/s 0.24.0, 0.23.1 [ 12317654, 12318883 ] 0.23.1, 0.24.0 [ 12318883, 12317654 ]
          Resolution Fixed [ 1 ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue breaks MAPREDUCE-3808 [ MAPREDUCE-3808 ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Allen Wittenauer made changes -
          Affects Version/s 2.0.0-alpha [ 12320354 ]
          Affects Version/s 0.24.0 [ 12317654 ]

            People

            • Assignee:
              Robert Joseph Evans
              Reporter:
              Siddharth Seth
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development