[SPARK-43378] SerializerHelper.deserializeFromChunkedBuffer leaks deserialization streams

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.4.1, 3.5.0
• Fix Version/s: 3.4.1, 3.5.0
• Component/s: Spark Core
• Assignee: Emil Ejbyfeldt
• Reporter: Emil Ejbyfeldt
• Created: 04/May/23 12:59
• Updated: 06/Jun/23 07:04
• Resolved: 05/May/23 00:34

Description

The method SerializerHelper.deserializeFromChunkedBuffer leaks serializations stream. This can lead to huge performance regressions when using kryo serializer as the spark application can become bottlenecked on the driver creating expensive kryo objects that are then leaked as part of the deserialization stream