[SPARK-48508] Client Side RPC optimization for Spark Connect

• Type: Umbrella
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect
• Assignee: Ruifeng Zheng
• Reporter: Ruifeng Zheng
• Created: 03/Jun/24 09:41
• Updated: 05/Jun/24 08:19

Issue Links

• links to: GitHub Pull Request #46848

Sub-Tasks

• Implement `StructType.treeString`: Resolved
• Make printSchema use the cached schema: Resolved
• Cache user specified schema in DataFrame.{to, mapInPandas, mapInArrow}: Resolved
• Cache user specified schema in applyInPandas and applyInArrow: Resolved
• Add test for cached schema: Resolved
• Cache more properties: Resolved
• Propagate cached schema in set operations: Resolved
• Propagate cached schema in dataframe operations: Resolved
• Should not cache `explain`: Resolved