[SPARK-37458] Remove unnecessary object serialization on foreachBatch

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 25/Nov/21 03:11
• Updated: 01/Dec/21 08:00
• Resolved: 01/Dec/21 08:00

Description

Currently, ForeachBatchSink leverages ExternalRDD with converting RDD[InternalRow] to RDD[T], to provide Dataset[T] to the user function. This adds SerializeFromObject in the plan, which is actually not required.

We can leverage LogicalRDD instead, to remove SerializeFromObject from the plan.

Issue Links

• links to: [Github] Pull Request #34706 (HeartSaVioR)