[YARN-8160] Yarn Service Upgrade: Support upgrade of service that use docker containers

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.2.0, 3.1.2
• Labels: Docker
• Target Version/s: 3.2.0
• Assignee: Chandni Singh
• Reporter: Chandni Singh
• Created: 13/Apr/18 21:35
• Updated: 16/Nov/18 21:04
• Resolved: 16/Nov/18 21:04

Description

Ability to upgrade dockerized yarn native services.
Ref: YARN-5637

Background
Container upgrade is supported by the NM via reInitializeContainer api. reInitializeContainer does NOT change the ContainerId of the upgraded container.
NM performs the following steps during reInitializeContainer:

• kills the existing process
• cleans up the container
• launches another container with the new ContainerLaunchContext

NOTE: ContainerLaunchContext holds all the information that needs to upgrade the container.

With reInitializeContainer, the following does NOT change

• container ID. This is not created by NM. It is provided to it and here RM is not creating another container allocation.
• localizedResources this stays the same if the upgrade does NOT require additional resources IIUC.

The following changes with reInitializeContainer

• the working directory of the upgraded container changes. It is NOT a relaunch.

Changes required in the case of docker container

• reInitializeContainer seems to not be working with Docker containers. Investigate and fix this.
• [Future change] Add an additional api to NM to pull the images and modify reInitializeContainer to trigger docker container launch without pulling the image first which could be based on a flag.
  • When the service upgrade is initialized, we can provide the user with an option to just pull the images on the NMs.
  • When a component instance is upgrade, it calls the reInitializeContainer with the flag pull-image set to false, since the NM will have already pulled the images.