[YARN-6798] Fix NM startup failure with old state store due to version mismatch

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha4
• Fix Version/s: 2.9.0, 3.0.0-beta1
• Component/s: nodemanager
• Hadoop Flags: Reviewed

Release Note

<!-- markdown -->

This fixes the LevelDB state store for the NodeManager. As of this patch, the state store versions now correspond to the following table.

* Previous Patch: YARN-5049
  * LevelDB Key: queued
  * Hadoop Versions: 2.9.0, 3.0.0-alpha1
  * Corresponding LevelDB Version: 1.2
* Previous Patch: YARN-6127
  * LevelDB Key: AMRMProxy/NextMasterKey
  * Hadoop Versions: 2.9.0, 3.0.0-alpha4
  * Corresponding LevelDB Version: 1.1

• Assignee: Botong Huang
• Reporter: Ray Chiang
• Created: 10/Jul/17 20:36
• Updated: 28/Aug/17 19:05
• Resolved: 18/Jul/17 19:40

Description

YARN-6703 rolled back the state store version number for the RM from 2.0 to 1.4.

YARN-6127 bumped the version for the NM to 3.0

private static final Version CURRENT_VERSION_INFO = Version.newInstance(3, 0);

YARN-5049 bumped the version for the NM to 2.0

private static final Version CURRENT_VERSION_INFO = Version.newInstance(2, 0);

During an upgrade, all NMs died after upgrading a C6 cluster from alpha2 to alpha4.

2017-07-07 15:48:17,259 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.service.ServiceStateException: java.io.IOException: Incompatible version for NM state: expecting NM state version 3.0, but loading version 2.0
        at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:246)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:307)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:748)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:809)
Caused by: java.io.IOException: Incompatible version for NM state: expecting NM state version 3.0, but loading version 2.0
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.checkVersion(NMLeveldbStateStoreService.java:1454)
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:1308)
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:307)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        ... 5 more
2017-07-07 15:48:17,277 INFO org.apache.hadoop.yarn.server.nodemanager.NodeManager: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NodeManager at xxx.gce.cloudera.com/aa.bb.cc.dd
************************************************************/

Issue Links

• is blocked by: YARN-5049 Extend NMStateStore to save queued container information
• is related to: YARN-6127 Add support for work preserving NM restart when AMRMProxy is enabled