[SPARK-11630] ClosureCleaner incorrectly warns for class based closures

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 1.4.1
• Fix Version/s: 2.4.0
• Component/s: Spark Core
• Assignee: Rekha Joshi
• Reporter: Frens Jan Rumph
• Created: 10/Nov/15 10:32
• Updated: 22/Jan/18 14:37
• Resolved: 22/Jan/18 14:37

Description

Spark's `ClosureCleaner` utility seems to check whether a function is an anonymous function: ClosureCleaner.scala on line 49 If not, it warns the user.

However, I'm using some class based functions. Something along the lines of:

trait FromUnreadRow[T] extends (UnreadRow => T) with Serializable

object ToPlainRow extends FromUnreadRow[PlainRow] {
  override def apply(row: UnreadRow): PlainRow = ???
}

This works just fine. I can't really see that the warning is actually useful in this case. I appreciate checking for common 'mistakes', but in my case a user might be alarmed unnecessarily.

Anything that can be done about this? Anything I can do?

Issue Links

• Is contained by: SPARK-14540 Support Scala 2.12 closures and Java 8 lambdas in ClosureCleaner
• links to: [Github] Pull Request #20337 (rekhajoshm)