[YARN-10442] RM should make sure node label file highly available

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.1
• Fix Version/s: 3.4.0
• Component/s: resourcemanager
• Assignee: Surendra Singh Lilhore
• Reporter: Surendra Singh Lilhore
• Created: 18/Sep/20 06:42
• Updated: 30/Oct/20 17:36
• Resolved: 30/Oct/20 17:35

Description

In one of my cluster RM failed transition to Active because node label file blocks are missing. I think RM should make sure important files are highly available . 

Caused by: com.google.protobuf.InvalidProtocolBufferException: Could not obtain block: BP-2121803626-10.0.0.22-1597301807397:blk_1073832522_91774 file=/yarn/node-labels/nodelabel.mirror
	at com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:238)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:253)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:259)
	at com.google.protobuf.AbstractParser.parseDelimitedFrom(AbstractParser.java:49)
	at org.apache.hadoop.yarn.proto.YarnServerResourceManagerServiceProtos$AddToClusterNodeLabelsRequestProto.parseDelimitedFrom(YarnServerResourceManagerServiceProtos.java:7493)
	at org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.loadFromMirror(FileSystemNodeLabelsStore.java:168)
	at org.apache.hadoop.yarn.nodelabels.FileSystemNodeLabelsStore.recover(FileSystemNodeLabelsStore.java:205)
	at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.initNodeLabelStore(CommonNodeLabelsManager.java:254)
	at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.serviceStart(CommonNodeLabelsManager.java:268)
	at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)(AbstractService.java:194)

Issue Links

• links to: Pull Request #2390