[CRUNCH-696] EOFException in App Master from FormatBundle#fromSerialized

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.1.0
• Component/s: Core
• Assignee: Josh Wills
• Reporter: Andrew Olson
• Created: 11/May/20 14:13
• Updated: 12/May/20 18:48
• Resolved: 12/May/20 18:48

Description

After CRUNCH-685 if there's a version mismatch between what a job was launched with and what it runs with in the cluster, a job can fail to start with an EOFException in the application master as shown below.

2019-10-11 14:36:41,327 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.io.EOFException

2019-10-11 14:36:41,332 ERROR [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.io.EOFException
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:531)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:511)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1614)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:511)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:301)
	at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$5.run(MRAppMaster.java:1572)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1569)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1502)
Caused by: java.lang.RuntimeException: java.io.EOFException
	at org.apache.crunch.io.FormatBundle.fromSerialized(FormatBundle.java:97)
	at org.apache.crunch.io.CrunchOutputs.getNamedOutputs(CrunchOutputs.java:127)
	at org.apache.crunch.io.CrunchOutputs.getOutputCommitter(CrunchOutputs.java:87)
	at org.apache.crunch.impl.mr.run.CrunchOutputFormat.getOutputCommitter(CrunchOutputFormat.java:52)
	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$2.call(MRAppMaster.java:529)
	... 11 more
Caused by: java.io.EOFException
	at java.io.DataInputStream.readBoolean(DataInputStream.java:244)
	at org.apache.crunch.io.FormatBundle.readFields(FormatBundle.java:292)
	at org.apache.crunch.io.FormatBundle.fromSerialized(FormatBundle.java:94)
	... 15 more

FormatBundle#fromSerialized should have handled the end of stream possibility when attempting to read the new boolean attribute introduced into the serialization.