Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-8901

Restart "NEVER" policy does not work with component dependency

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 3.1.1
    • 3.3.0, 3.2.1, 3.1.3
    • None
    • None

    Description

      Scenario:

      1) Launch an application with two components. master and worker. Here, worker is dependent on master. ( Worker should be launched only after master is launched )
      2) Set restart_policy = NEVER for both master and worker.

      sample launch.json
      {
      	"name": "mawo-hadoop-ut",
              "artifact": {
                      "type": "DOCKER",
                      "id": "xxx"
              },
              "configuration": {
                      "env": {
                             "YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK": "hadoop"
                       },
                      "properties": {
                             "docker.network": "hadoop"
                      }
              },
      	"components": [{
      		"dependencies": [],
      		"resource": {
      			"memory": "2048",
      			"cpus": "1"
      		},
      		"name": "master",
                      "run_privileged_container": true,
      		"number_of_containers": 1,
      		"launch_command": "start master",
                      "restart_policy": "NEVER",
      	}, {
      		"dependencies": ["master"],
      		"resource": {
      			"memory": "8072",
      			"cpus": "1"
      		},
      		"name": "worker",
                      "run_privileged_container": true,
      		"number_of_containers": 10,
      		"launch_command": "start worker",
                      "restart_policy": "NEVER",
      	}],
      	"lifetime": -1,
      	"version": 1.0
      }

      When restart policy is selected to NEVER, AM never launches Worker component. It get stuck with below message.

      2018-10-17 15:11:58,560 [Component  dispatcher] INFO  component.Component - [COMPONENT master] Transitioned from FLEXING to STABLE on CHECK_STABLE event.
      2018-10-17 15:11:58,560 [pool-7-thread-1] INFO  instance.ComponentInstance - [COMPINSTANCE master-0 : container_e41_1539027682947_0020_01_000002] Transitioned from STARTED to READY on BECOME_READY event
      2018-10-17 15:11:58,560 [pool-7-thread-1] INFO  component.Component - [COMPONENT worker]: Dependency master not satisfied, only 1 of 1 instances are ready or the dependent component has not completed 
      2018-10-17 15:12:28,556 [pool-7-thread-1] INFO  component.Component - [COMPONENT worker]: Dependency master not satisfied, only 1 of 1 instances are ready or the dependent component has not completed 
      2018-10-17 15:12:58,556 [pool-7-thread-1] INFO  component.Component - [COMPONENT worker]: Dependency master not satisfied, only 1 of 1 instances are ready or the dependent component has not completed 
      2018-10-17 15:13:28,556 [pool-7-thread-1] INFO  component.Component - [COMPONENT worker]: Dependency master not satisfied, only 1 of 1 instances are ready or the dependent component has not completed 
      2018-10-17 15:13:58,556 [pool-7-thread-1] INFO  component.Component - [COMPONENT worker]: Dependency master not satisfied, only 1 of 1 instances are ready or the dependent component has not completed 
      2018-10-17 15:14:28,556 [pool-7-thread-1] INFO  component.Component - [COMPONENT worker]: Dependency master not satisfied, only 1 of 1 instances are ready or the dependent component has not completed 

      'NEVER' restart policy expects master component to be finished before starting workers. Master component can not finish the job without workers. Thus, it create a deadlock.

      The logic for 'NEVER' restart policy should be fixed to allow worker components to be launched as soon as master component is in READY state.

      Attachments

        1. YARN-8901.1.patch
          5 kB
          Suma Shivaprasad
        2. YARN-8901.2.patch
          7 kB
          Suma Shivaprasad
        3. YARN-8901.3.patch
          7 kB
          Suma Shivaprasad

        Activity

          People

            suma.shivaprasad Suma Shivaprasad
            yeshavora Yesha Vora
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: