[YARN-9247] NM fails to start when downgrading from 3.x to 2.7

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.2
• Component/s: yarn
• Assignee: Jiandan Yang
• Reporter: Jiandan Yang
• Created: 29/Jan/19 02:04
• Updated: 08/Feb/21 15:08

Description

NM will throws java.io.IOException: Unexpected container state key when downgrading from 3.1.3 to 2.7.2 duo to starttime added by hadoop3.x is not known by old version.

2019-01-28 12:12:05,227 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager
org.apache.hadoop.service.ServiceStateException: java.io.IOException: Unexpected container state key: ContainerManager/containers/container_e42_1548641704310_0002_01_000001/starttime
        at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:263)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:536)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:584)
Caused by: java.io.IOException: Unexpected container state key: ContainerManager/containers/container_e42_1548641704310_0002_01_000001/starttime
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.loadContainerState(NMLeveldbStateStoreService.java:242)
        at org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.loadContainersState(NMLeveldbStateStoreService.java:178)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:290)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:274)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        ... 5 more