[MAPREDUCE-4150] Versioning and rolling upgrades for MR2

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.23.1
• Component/s: mrv2
• Assignee: Unassigned
• Reporter: Ahmed Radwan
• Created: 13/Apr/12 12:36
• Updated: 28/Dec/14 21:31

Description

It doesn't seem that Yarn components, for example the ResourceManager or NodeManager, do build/package version checking before trying to communicate with each other.

The objective of this ticket is to support the following requirements / use cases:

• New versions can be marked incompatible with old versions, and services should be prevented from communicating with each other in such case. This will avoid non-deterministic behavior/problems resulting from incompatible components trying to communicate with each other.

• Permitting a policy for running different - but compatible - versions on the same cluster (for example, in a rolling upgrade scenario). See HDFS-2983 for the corresponding HDFS implementation.

Issue Links

• depends upon: HADOOP-8280 Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common.
• is related to: YARN-819 ResourceManager and NodeManager should check for a minimum allowed version
• is related to: MAPREDUCE-5831 Old MR client is not compatible with new MR application
• relates to: HDFS-2983 Relax the build version check to permit rolling upgrades within a release
• relates to: YARN-666 [Umbrella] Support rolling upgrades in YARN

Sub-Tasks

• Run MapReduce framework via the distributed cache: Closed
• ShuffleHandler DB schema should be versioned with compatible/incompatible changes: Closed
• Improved shuffle error handling across NM restarts: Closed