Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
1.0.0
-
None
Description
Using SparkConf with SparkContext as described in the Programming Guide does NOT work in Python:
conf = SparkConf.setAppName("blah")
sc = SparkContext(conf)
When I tried I got
AttributeError: 'SparkConf' object has no attribute '_get_object_id'
[This equivalent code in Scala works fine:
val conf = new SparkConf().setAppName("blah")
val sc = new SparkContext(conf)]
I think this is because there's no equivalent for the Scala constructor SparkContext(SparkConf).
Workaround:
If I explicitly set the conf parameter in the python call, it does work:
sconf = SparkConf.setAppName("blah")
sc = SparkContext(conf=sconf)