[SPARK-4170] Closure problems when running Scala app that "extends App"

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.1.0
• Component/s: Spark Core
• Assignee: Sean R. Owen
• Reporter: Sean R. Owen
• Created: 31/Oct/14 09:55
• Updated: 23/Feb/19 03:19
• Resolved: 27/Nov/14 17:08

Description

Michael Albert noted this problem on the mailing list (http://apache-spark-user-list.1001560.n3.nabble.com/BUG-when-running-as-quot-extends-App-quot-closures-don-t-capture-variables-td17675.html):

object DemoBug extends App {
    val conf = new SparkConf()
    val sc = new SparkContext(conf)

    val rdd = sc.parallelize(List("A","B","C","D"))
    val str1 = "A"

    val rslt1 = rdd.filter(x => { x != "A" }).count
    val rslt2 = rdd.filter(x => { str1 != null && x != "A" }).count
    
    println("DemoBug: rslt1 = " + rslt1 + " rslt2 = " + rslt2)
}

This produces the output:

DemoBug: rslt1 = 3 rslt2 = 0

If instead there is a proper "main()", it works as expected.

I also this week noticed that in a program which "extends App", some values were inexplicably null in a closure. When changing to use main(), it was fine.

I assume there is a problem with variables not being added to the closure when main() doesn't appear in the standard way.

Issue Links

• is duplicated by: SPARK-2175 Null values when using App trait.
• links to: [Github] Pull Request #3497 (srowen)
• links to: GitHub Pull Request #3497