[TEZ-2367] Corruption of TezHeartbeatRequest

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Duplicate
• Affects Version/s: 0.7.0
• Target Version/s: 0.7.0
• Assignee: Bikas Saha
• Reporter: Siddharth Seth
• Created: 24/Apr/15 20:39
• Updated: 24/Apr/15 22:46
• Resolved: 24/Apr/15 22:32

Description

The following exception is seen in the AM logs while attempting to deserialize a heartbeat request.

java.lang.ArrayIndexOutOfBoundsException: 1382376565
        at org.apache.tez.runtime.api.impl.EventMetaData.readFields(EventMetaData.java:120)
        at org.apache.tez.runtime.api.impl.TezEvent.readFields(TezEvent.java:271)
        at org.apache.tez.runtime.api.impl.TezHeartbeatRequest.readFields(TezHeartbeatRequest.java:110)
        at org.apache.hadoop.io.ObjectWritable.readObject(ObjectWritable.java:285)
        at org.apache.hadoop.ipc.WritableRpcEngine$Invocation.readFields(WritableRpcEngine.java:160)
        at org.apache.hadoop.ipc.Server$Connection.processRpcRequest(Server.java:1869)
        at org.apache.hadoop.ipc.Server$Connection.processOneRpc(Server.java:1801)
        at org.apache.hadoop.ipc.Server$Connection.readAndProcess(Server.java:1559)
        at org.apache.hadoop.ipc.Server$Listener.doRead(Server.java:784)
        at org.apache.hadoop.ipc.Server$Listener$Reader.doRunLoop(Server.java:650)
        at org.apache.hadoop.ipc.Server$Listener$Reader.run(Server.java:621)

TEZ-2234 is what changed the serialization most recently. bikassaha - mind taking a look.
From a quick glance, it looks like this is caused by the way TaskStatistics are serialized. ioStatistics.size followed by an iterator over ioStatistics.
ioStatistics can change during this time as different Inputs / Outputs get initialized. Synchronizing should fix this.

Also, setting the statistics may require synchronization to ensure correct values are written.

Issue Links

• duplicates: TEZ-2314 Tez task attempt failures due to bad event serialization