[FLINK-10469] FileChannel may not write the whole buffer in a single call to FileChannel.write(Buffer buffer)

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.4.1, 1.4.2, 1.5.3, 1.5.4, 1.6.0, 1.6.1, 1.6.2, 1.7.0
• Fix Version/s: 1.5.5, 1.6.2, 1.7.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Nico Kruber
• Reporter: Yun Gao
• Created: 01/Oct/18 05:22
• Updated: 28/Feb/19 13:20
• Resolved: 09/Oct/18 16:17

Description

Currently all the calls to FileChannel.write(ByteBuffer src) assumes that this method will not return before the whole buffer is written, like the one in AsynchronousFileIOChannel.write().

 

However, this assumption may not be right for all the environments. We have encountered the case that only part of a buffer was written on a cluster with a high IO load, and the target file got messy. 

 

To fix this issue, I think we should add a utility method in the org.apache.flink.util.IOUtils to ensure the whole buffer is written with a loop，and replace all the calls to FileChannel.write(ByteBuffer) with this new method. 

Issue Links

• causes: FLINK-10682 EOFException occurs during deserialization of Avro class
• links to: GitHub Pull Request #6788