[SLIDER-787] App Upgrade/Reconfig support in Slider

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Slider 0.61
• Fix Version/s: Slider 0.80
• Component/s: app-package, appmaster
• Sprint: Slider April #1
• Assignee: Gour Saha
• Reporter: Gour Saha
• Created: 18/Feb/15 00:28
• Updated: 19/Dec/15 13:08
• Resolved: 28/Apr/15 20:57

Description

Terminologies:
Here "App" means the application that is deployed by Slider e.g. HBase, Storm, Accumulo, Memcached, etc.

Use-case:

• YARN/HDFS/Zookeeper Upgrade
• YARN/HDFS/Zookeeper reconfiguration
• Slider Upgrade
• App Upgrade
• App Reconfiguration

YARN/HDFS Upgrade:
Upgrade of YARN/HDFS underlying infra, libraries, etc.

YARN/HDFS reconfiguration:
Reconfiguration of infrastructure components which Slider relies on.

Slider Upgrade:
Today you can do it by stopping the application and use the new version of the Slider to start the application.

Slider Upgrade translates to the upgrade of:

• Slider AppMaster
• Slider Client
• Slider Agent

App Upgrade

• Support manual upgrade of Slider application
• Provide individual component specific upgrade [ Nice to Have ]
• Need to allow rollback

Steps to upgrade:

• Stop app
• slider set-app-package app --package <new package>
• Start app

Rollback:

• Stop app
• slider set-app-package app --package <old package>
• Start app

New component instances should be tried at best to be created in same host and port [ strong preference ] including AM. This task should be possible through CLI and new REST API.

Rolling upgrade:

• Slider CLI will not provide component upgrade ordering and start/stop - needs to be orchestrated by external scripts

App Reconfiguration

• Allow updating application configurations (appConfig.json or resources.json)
• Allow rolling restart of all containers to have the new configuration take effect

Slider needs to implement

• Need specific container stop/start ability
• Need upgrade package functionality [ required for in-place upgrade too ]
• Need update capability for configuration
• Component and AM host port to be preserved in best-effort

Backward Compatibility

• New apps "may" have new configuration based on new packaging. So upgrade needs to take care of that.
• Reconfiguration will NOT require to support new style of configuration metadata.

Issue Links

• duplicates: SLIDER-226 Provide mechanism for rolling upgrade
• is duplicated by: SLIDER-290 Uber JIRA: Slider upgrade support
• is related to: YARN-4470 [Umbrella] Application Master in-place upgrade

Sub-Tasks

• Expose client API to list containers for a running application with options to filter by role: Resolved
• Fix SLIDER-827 - Slider package --instances does not list any applications: Resolved
• Introduce versioning in Slider tarball application packages: Resolved
• Core coding to introduce upgrade related operations and all dependencies: Resolved
• Introduce versioning in Slider AM and agents to provide Slider AM capability to track by version, facilitating rolling upgrade and downgrade/rollback: Resolved
• More tests: Resolved
• Site documentation for upgrade/reconfig support in Slider: Resolved