[MAPREDUCE-7290] ShuffleHeader should be compatible between client&server when rolling update

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.6.0
• Fix Version/s: 2.9.0, 3.3.0
• Component/s: mrv2
• Labels: pull-request-available
• Target Version/s: 3.3.9
• Assignee: Unassigned
• Reporter: chaoli
• Created: 07/Aug/20 03:39
• Updated: 29/Nov/22 15:56

Description

Recently, we need to add a property in ShuffleHeader class. And there were a problem that client&server may not compatible because of serialization/deserialization when we were rolling update NodeManager instance. 

To solve the problem, we made an improvement to help shuffle header version upgrade smoothly. 

In HeaderVersion class, shuffle client uses targetVersion(eg 1.2.0) to negotiate with shuffle server. In that case, shuffle server has three cases:

1. shuffle server does not have the newest code which is able to support version chosen, and we have to use defaultVersion(eg 1.0.0) instead*.*
2. shuffle server can support older version(eg 1.1.0), then it can reply to shuffle client with compatibleVersion(eg 1.1.0), and finally, client and server can use the same version.
3. shuffle server can support older version(eg 1.3.0), then it can reply to shuffle client with compatibleVersion(eg 1.2.0), and finally, client and server can use the same version.

Then, client and server can serialize&deserialize property using the available version.

 

/**
 * unified hard code header version for new properties.
 * @throws IOException
 */
private void writeByVersion(DataOutput out) throws IOException {
  if (headerVersion == null || headerVersion.compareTo(DEFAULT_HEADER_VERSION_INSTANCE) == 0) {
    return;
  }
  // if current version larger then target version,
  // we should serialize the properties owned by the version in order.
  if (headerVersion.compareTo(HEADER_VERSION_INSTANCE_V1_1) >= 0) {
    // here write version properties
  }
}

 

 

Issue Links

• links to: GitHub Pull Request #2200