[FLINK-256] Nephele "loses" data packets

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: pre-apache
• Labels: github-import
• Assignee: Ufuk Celebi
• Reporter: GitHub Import
• Created: 09/Jun/14 11:50
• Updated: 13/Jun/14 00:06
• Resolved: 13/Jun/14 00:06

Description

A pact job dies with the following message: "An error occurred in the channel: Expected data packet 25 but received 27"

The error occurred both in version 0.21 and with the current 0.4-snapshot. Apparently only with bigger data sizes. 15GB Input data worked fine, 22GB crashed. I sent the job, that caused the error to Stephan Ewen.

Full stack trace:
15:32:11,639 ERROR eu.stratosphere.pact.runtime.task.RegularPactTask - Error in PACT code: Join tweets and dates (2/4)
15:32:11,640 ERROR eu.stratosphere.pact.runtime.task.RegularPactTask - java.io.IOException: An error occurred in the channel: Expected data packet 25 but received 27
java.io.IOException: An error occurred in the channel: Expected data packet 25 but received 27
at eu.stratosphere.nephele.io.channels.bytebuffered.AbstractByteBufferedInputChannel.isClosed(AbstractByteBufferedInputChannel.java:144)
at eu.stratosphere.nephele.io.RuntimeInputGate.isClosed(RuntimeInputGate.java:261)
at eu.stratosphere.nephele.io.RuntimeInputGate.readRecord(RuntimeInputGate.java:182)
at eu.stratosphere.nephele.io.MutableRecordReader.next(MutableRecordReader.java:80)
at eu.stratosphere.pact.runtime.task.util.PactRecordNepheleReaderIterator.next(PactRecordNepheleReaderIterator.java:62)
at eu.stratosphere.pact.runtime.task.util.PactRecordNepheleReaderIterator.next(PactRecordNepheleReaderIterator.java:27)
at eu.stratosphere.pact.runtime.hash.MutableHashTable$ProbeIterator.next(MutableHashTable.java:1525)
at eu.stratosphere.pact.runtime.hash.MutableHashTable.processProbeIter(MutableHashTable.java:450)
at eu.stratosphere.pact.runtime.hash.MutableHashTable.nextRecord(MutableHashTable.java:536)
at eu.stratosphere.pact.runtime.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:116)
at eu.stratosphere.pact.runtime.task.MatchDriver.run(MatchDriver.java:164)
at eu.stratosphere.pact.runtime.task.RegularPactTask.run(RegularPactTask.java:372)
at eu.stratosphere.pact.runtime.task.RegularPactTask.invoke(RegularPactTask.java:291)
at eu.stratosphere.nephele.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:344)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Expected data packet 25 but received 27
at eu.stratosphere.nephele.taskmanager.runtime.RuntimeInputChannelContext.queueTransferEnvelope(RuntimeInputChannelContext.java:148)
at eu.stratosphere.nephele.taskmanager.bytebuffered.ByteBufferedChannelManager.processEnvelopeWithBuffer(ByteBufferedChannelManager.java:365)
at eu.stratosphere.nephele.taskmanager.bytebuffered.ByteBufferedChannelManager.processEnvelope(ByteBufferedChannelManager.java:331)
at eu.stratosphere.nephele.taskmanager.bytebuffered.ByteBufferedChannelManager.processEnvelopeFromNetwork(ByteBufferedChannelManager.java:644)
at eu.stratosphere.nephele.taskmanager.bytebuffered.IncomingConnection.read(IncomingConnection.java:100)
at eu.stratosphere.nephele.taskmanager.bytebuffered.IncomingConnectionThread.doRead(IncomingConnectionThread.java:187)
at eu.stratosphere.nephele.taskmanager.bytebuffered.IncomingConnectionThread.run(IncomingConnectionThread.java:126)
15:32:12,362 WARN eu.stratosphere.pact.runtime.task.RegularPactTask - Cancelling PACT code: Join tweets and dates (2/4)
15:32:12,362 INFO eu.stratosphere.nephele.execution.ExecutionStateTransition - TM: ExecutionState set from RUNNING to FAILED for task Join tweets and dates (2/4)
15:32:12,362 ERROR eu.stratosphere.nephele.taskmanager.runtime.RuntimeTask - java.io.IOException: An error occurred in the channel: Expected data packet 25 but received 27
at eu.stratosphere.nephele.io.channels.bytebuffered.AbstractByteBufferedInputChannel.isClosed(AbstractByteBufferedInputChannel.java:144)
at eu.stratosphere.nephele.io.RuntimeInputGate.isClosed(RuntimeInputGate.java:261)
at eu.stratosphere.nephele.io.RuntimeInputGate.readRecord(RuntimeInputGate.java:182)
at eu.stratosphere.nephele.io.MutableRecordReader.next(MutableRecordReader.java:80)
at eu.stratosphere.pact.runtime.task.util.PactRecordNepheleReaderIterator.next(PactRecordNepheleReaderIterator.java:62)
at eu.stratosphere.pact.runtime.task.util.PactRecordNepheleReaderIterator.next(PactRecordNepheleReaderIterator.java:27)
at eu.stratosphere.pact.runtime.hash.MutableHashTable$ProbeIterator.next(MutableHashTable.java:1525)
at eu.stratosphere.pact.runtime.hash.MutableHashTable.processProbeIter(MutableHashTable.java:450)
at eu.stratosphere.pact.runtime.hash.MutableHashTable.nextRecord(MutableHashTable.java:536)
at eu.stratosphere.pact.runtime.hash.BuildFirstHashMatchIterator.callWithNextKey(BuildFirstHashMatchIterator.java:116)
at eu.stratosphere.pact.runtime.task.MatchDriver.run(MatchDriver.java:164)
at eu.stratosphere.pact.runtime.task.RegularPactTask.run(RegularPactTask.java:372)
at eu.stratosphere.pact.runtime.task.RegularPactTask.invoke(RegularPactTask.java:291)
at eu.stratosphere.nephele.execution.RuntimeEnvironment.run(RuntimeEnvironment.java:344)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Expected data packet 25 but received 27
at eu.stratosphere.nephele.taskmanager.runtime.RuntimeInputChannelContext.queueTransferEnvelope(RuntimeInputChannelContext.java:148)
at eu.stratosphere.nephele.taskmanager.bytebuffered.ByteBufferedChannelManager.processEnvelopeWithBuffer(ByteBufferedChannelManager.java:365)
at eu.stratosphere.nephele.taskmanager.bytebuffered.ByteBufferedChannelManager.processEnvelope(ByteBufferedChannelManager.java:331)
at eu.stratosphere.nephele.taskmanager.bytebuffered.ByteBufferedChannelManager.processEnvelopeFromNetwork(ByteBufferedChannelManager.java:644)
at eu.stratosphere.nephele.taskmanager.bytebuffered.IncomingConnection.read(IncomingConnection.java:100)
at eu.stratosphere.nephele.taskmanager.bytebuffered.IncomingConnectionThread.doRead(IncomingConnectionThread.java:187)
at eu.stratosphere.nephele.taskmanager.bytebuffered.IncomingConnectionThread.run(IncomingConnectionThread.java:126)

15:32:12,394 INFO eu.stratosphere.nephele.taskmanager.runtime.RuntimeTask - Canceling Hashtag Polarity Match (2/4)

---------------- Imported from GitHub ----------------
Url: https://github.com/stratosphere/stratosphere/issues/256
Created by: matpeters
Labels: bug, runtime, user satisfaction,
Created at: Fri Nov 08 13:57:57 CET 2013
State: open