Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5617

map task is not re-launched when the task is failed while reducers are running with full cluster capacity - which will lead to job hang

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Invalid
    • Affects Version/s: 2.2.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      SuSe Linux

      Description

      In a Cluster with 16GB capacity, job has started with 100maps and 10 reducers.

      When the reducers has started its execution, one NM has went down and resulted a failure for 2 maps. But at this time, remaining 8Gb was used by 6 reducers and AM. So there was no place to launch the failed maps. [NM never came up again, and cluster size became 8GB]

      If we kill one of reducers, then also the map cannot be launched as the priority of Failed map is lesser than that of reducer. So the remaining reducer only will get allocated from RM side.

      This is causing a hang for in reducer side.

        Activity

        Hide
        Vinod Kumar Vavilapalli added a comment -

        Sunil G,

        If we kill one of reducers, then also the map cannot be launched as the priority of Failed map is lesser than that of reducer. So the remaining reducer only will get allocated from RM side. This is causing a hang for in reducer side.

        This isn't correctly interpreted. I can understand it being not straight-forward, but lower numbered priority of failed map actually means it is of higher priority w.r.t scheduling.

        You may have been seeing something else. If you think this is a stale issue, please close this as invalid. Thanks.

        Show
        Vinod Kumar Vavilapalli added a comment - Sunil G , If we kill one of reducers, then also the map cannot be launched as the priority of Failed map is lesser than that of reducer. So the remaining reducer only will get allocated from RM side. This is causing a hang for in reducer side. This isn't correctly interpreted. I can understand it being not straight-forward, but lower numbered priority of failed map actually means it is of higher priority w.r.t scheduling. You may have been seeing something else. If you think this is a stale issue, please close this as invalid. Thanks.
        Hide
        Sunil G added a comment -

        Yes Vinod.

        I am marking this as invalid. I checked a similar scenario, and i could see map task is getting re-launched.

        However, I will check more detail., And if its reoccurring, i ll reopen.

        Show
        Sunil G added a comment - Yes Vinod. I am marking this as invalid. I checked a similar scenario, and i could see map task is getting re-launched. However, I will check more detail., And if its reoccurring, i ll reopen.

          People

          • Assignee:
            Unassigned
            Reporter:
            Sunil G
          • Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development