[SPARK-30264] Unexpected behaviour when using persist MEMORY_ONLY in RDD

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.0, 2.4.4
• Component/s: Java API
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: moshe ohaion
• Created: 15/Dec/19 13:08
• Updated: 25/May/21 01:54
• Resolved: 25/May/21 01:43

Description

Persist method with MEMORY_ONLY behave different than using with MEMORY_ONLY_SER.

persist(StorageLevel.MEMORY_ONLY()).distinct().count() return 1

while persist(StorageLevel.MEMORY_ONLY_SER()).distinct().count() return 100

I expect both to return the same results. The right result is 100, for some reason MEMORY_ONLY causing all the objects in the RDD to be the same one.