Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-1996

Provide alternative policies for UNHEALTHY nodes.

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.4.0
    • None
    • nodemanager, scheduler
    • None

    Description

      Currently, UNHEALTHY nodes can significantly prolong execution of large expensive jobs as demonstrated by MAPREDUCE-5817, and downgrade the cluster health even further due to positive feedback. A container set that might have deemed the node unhealthy in the first place starts spreading across the cluster because the current node is declared unusable and all its containers are killed and rescheduled on different nodes.

      To mitigate this, we experiment with a patch that allows containers already running on a node turning UNHEALTHY to complete (drain) whereas no new container can be assigned to it until it turns healthy again.

      This mechanism can also be used for graceful decommissioning of NM. To this end, we have to write a health script such that it can deterministically report UNHEALTHY. For example with

      if [ -e $1 ] ; then                                                                
        echo ERROR Node decommmissioning via health script hack                          
      fi 
      

      In the current version patch, the behavior is controlled by a boolean property yarn.nodemanager.unhealthy.drain.containers. More versatile policies are possible in the future work. Currently, the health state of a node is binary determined based on the disk checker and the health script ERROR outputs. However, we can as well interpret health script output similar to java logging levels (one of which is ERROR) such as WARN, FATAL. Each level can then be treated differently. E.g.,

      • FATAL: unusable like today
      • ERROR: drain
      • WARN: halve the node capacity.
        complimented with some equivalence rules such as 3 WARN messages == ERROR, 2*ERROR == FATAL, etc.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jira.shegalov Gera Shegalov Assign to me
            jira.shegalov Gera Shegalov

            Dates

              Created:
              Updated:

              Slack

                Issue deployment