[SPARK-17491] MemoryStore.putIteratorAsBytes() may silently lose values when KryoSerializer is used

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.1, 2.1.0
• Component/s: Block Manager, Spark Core
• Labels: correctness
• Target Version/s: 2.0.1
• Assignee: Josh Rosen
• Reporter: Josh Rosen
• Created: 10/Sep/16 06:47
• Updated: 17/May/20 18:21
• Resolved: 17/Sep/16 18:47

Description

MemoryStore.putIteratorAsBytes() may silently lose values when used with KryoSerializer because it does not properly close the serialization stream before attempting to deserialize the already-serialized values, which may cause values buffered in Kryo's internal buffers to not be read.

This is the root cause behind a user-reported "wrong answer" bug in PySpark caching reported by Ben Leslie on the Spark user mailing list in a thread titled "pyspark persist MEMORY_ONLY vs MEMORY_AND_DISK")

Issue Links

• links to: [Github] Pull Request #15043 (JoshRosen)