[SPARK-40163] [SPARK][SQL] feat: SparkSession.confing(Map)

• Type: Improvement
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: Spark Core
• Target Version/s: 3.4.0
• Assignee: seunggabi
• Reporter: seunggabi
• Created: 21/Aug/22 08:45
• Updated: 02/Sep/22 13:48
• Resolved: 21/Aug/22 19:29

Description

https://github.com/apache/spark/pull/37478 

• as-is

  private fun config(builder: SparkSession.Builder): SparkSession.Builder {
          val map = YamlUtils.read(this::class.java, "spark", Extension.YAML)
  
          var b = builder
          map.keys.forEach {
              val k = it
              val v = map[k]
  
              b = when (v) {
                  is Long -> b.config(k, v)
                  is String -> b.config(k, v)
                  is Double -> b.config(k, v)
                  is Boolean -> b.config(k, v)
                  else -> b
              }
          }
  
          return b
      }
  } 

• to-be

  private fun config(builder: SparkSession.Builder): SparkSession.Builder {
          val map = YamlUtils.read(this::class.java, "spark", Extension.YAML)
  
          return b.config(map)
      }
  } 

Issue Links

• links to: [Github] Pull Request #37478 (seunggabi)
• links to: [Github] Pull Request #37772 (LuciferYang)
• links to: [Github] Pull Request #37772 (LuciferYang)