[SPARK-2003] SparkContext(SparkConf) doesn't work in pyspark

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.1, 1.1.0
• Component/s: Documentation, PySpark
• Assignee: Unassigned
• Reporter: Diana Carroll
• Created: 03/Jun/14 19:08
• Updated: 02/Jul/14 00:21
• Resolved: 02/Jul/14 00:21

Description

Using SparkConf with SparkContext as described in the Programming Guide does NOT work in Python:
conf = SparkConf.setAppName("blah")
sc = SparkContext(conf)
When I tried I got
AttributeError: 'SparkConf' object has no attribute '_get_object_id'

[This equivalent code in Scala works fine:
val conf = new SparkConf().setAppName("blah")
val sc = new SparkContext(conf)]

I think this is because there's no equivalent for the Scala constructor SparkContext(SparkConf).

Workaround:
If I explicitly set the conf parameter in the python call, it does work:
sconf = SparkConf.setAppName("blah")
sc = SparkContext(conf=sconf)