[SPARK-45136] Improve ClosureCleaner to support closures defined in Ammonite REPL

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0, 3.5.1
• Fix Version/s: 4.0.0
• Component/s: Connect
• Labels: pull-request-available
• Assignee: Vsevolod Stepanov
• Reporter: Vsevolod Stepanov
• Created: 12/Sep/23 14:13
• Updated: 26/Oct/23 01:38
• Resolved: 26/Oct/23 01:38

Description

ConnectRepl uses Ammonite REPL with  CodeClassWrapper to run Scala code. It means that each code cell is wrapped into a separate object. If there are multiple variables defined in the same cell / code block it will lead to capturing extra variables, increasing serialized UDF payload size or making it non-serializable.

For example, this code

// cell 1 
{
  val x = 100
  val y = new NonSerializable
}

// cell 2
spark.range(10).map(i => i + x).agg(sum("value")).collect()

will fail because lambda will capture both `x` and `y` as they're defined in the same wrapper object

Issue Links

• links to: GitHub Pull Request #42995