[MRQL-51] Support for Spark 1.1.0

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.9.4
• Component/s: Run-Time/Spark
• Assignee: Leonidas Fegaras
• Reporter: Leonidas Fegaras
• Created: 23/Sep/14 20:24
• Updated: 23/Sep/14 20:27
• Resolved: 23/Sep/14 20:27

Description

The following patch makes some necessary changes to support Spark 1.1.0. The Spark evaluator now uses RDD.mapPartitionsWithIndex to extract RDD elements lazily (as needed) instead of using the internal Spark method RDD.iterator, which needs access to TaskContext. It also changes the JavaSparkContext to read the Spark config parameters.