[SPARK-23706] spark.conf.get(value, default=None) should produce None in PySpark

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.3.1, 2.4.0
• Component/s: PySpark
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 16/Mar/18 04:03
• Updated: 12/Dec/22 18:10
• Resolved: 18/Mar/18 11:30

Description

Scala:

scala> spark.conf.get("hey")
java.util.NoSuchElementException: hey
  at org.apache.spark.sql.internal.SQLConf$$anonfun$getConfString$2.apply(SQLConf.scala:1600)
  at org.apache.spark.sql.internal.SQLConf$$anonfun$getConfString$2.apply(SQLConf.scala:1600)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.sql.internal.SQLConf.getConfString(SQLConf.scala:1600)
  at org.apache.spark.sql.RuntimeConfig.get(RuntimeConfig.scala:74)
  ... 49 elided

scala> spark.conf.get("hey", null)
res1: String = null

scala> spark.conf.get("spark.sql.sources.partitionOverwriteMode", null)
res2: String = null

Python:

>>> spark.conf.get("hey")
...
py4j.protocol.Py4JJavaError: An error occurred while calling o30.get.
: java.util.NoSuchElementException: hey
...

>>> spark.conf.get("hey", None)
...
py4j.protocol.Py4JJavaError: An error occurred while calling o30.get.
: java.util.NoSuchElementException: hey
...

>>> spark.conf.get("spark.sql.sources.partitionOverwriteMode", None)
u'STATIC'

Issue Links

• links to: [Github] Pull Request #20841 (HyukjinKwon)