[FLINK-5908] Blob Cache can (rarely) get corrupted on failed blob downloads

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.3.0, 1.4.0
• Fix Version/s: 1.4.0
• Component/s: Runtime / Coordination, Runtime / Network
• Assignee: Nico Kruber
• Reporter: Stephan Ewen
• Created: 24/Feb/17 16:37
• Updated: 06/Oct/17 09:47
• Resolved: 06/Oct/17 09:47

Description

The Blob Cache downloads files directly to the target file location.
While it tries to clean up failed attempts, there is a change that this cleanup does not complete.

In that case, we have a corrupt file at the target location. The blob cache then assumes that it already has the file cached already and future requests do not attempt to re-download the file.

The fix would be to download to a temp file name, validate the integrity, and rename to the target file path when the validation succeeds.

The validation for "content addressable" could even include validating the checksum hash.

Issue Links

• is related to: FLINK-6916 FLIP-19: Improved BLOB storage architecture
• is superceded by: FLINK-7068 change BlobService sub-classes for permanent and transient BLOBs