Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6982

Containers on lost nodes are considered failed after a too long time.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Duplicate
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: mr-am
    • Labels:
      None
    • Environment:

      cdh5.5.0

      Description

      Containers on lost nodes (nodemanager being unavailable or server being unavailable) are considered failed after a too long time.
      This is due to the AppMaster trying to cleanup the container on the unavailable node.
      The proposed path will limit the impact of this timeout by managing NodeManager lost events on AM as described below:

      • on nodemanager service unavailibility (crash, oom ...):
        When receiving lost NodeManager events, it failed the impacted attempt and do not go through the cleanup stage.
      • on nodemanager server unavailibility with default settings AM detect first that the attempt is in timeout and try to cleanup the attempt:
        When receiving lost NodeManager events, it stop the cleanup process on the impacted container and failed the attempt.

      This reduce the duration of the timeout to the timeout for detecting a NodeManager down.

      Similar issue than MAPREDUCE-6659 on which I can't attached the patch.

        Attachments

        1. MAPREDUCE-6982.patch
          13 kB
          Nicolas Fraison

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                nfraison.criteo Nicolas Fraison
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: