[SPARK-49673] Increase maxBatchSize for Connect's sqlCommandResult

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect
• Labels: pull-request-available
• Target Version/s: 4.0.0
• Epic Link: Spark Connect
• Assignee: Robert Dillitz
• Reporter: Robert Dillitz
• Created: 16/Sep/24 15:29
• Updated: 18/Sep/24 23:54
• Resolved: 18/Sep/24 23:54

Description

Increase the default maxBatchSize from 4MiB * 0.7 to 128MiB (=
CONNECT_GRPC_MAX_MESSAGE_SIZE) * 0.7 when creating the single Arrow batch for the SqlCommandResult in the SparkConnectPlanner. This lets us return much larger LocalRelations in the SqlCommandResult (for example for the SHOW PARTITIONS command) while still staying within the GRPC message size limit.

Issue Links

• links to: GitHub Pull Request #48122