[SPARK-22533] SparkConfigProvider does not handle deprecated config keys

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Spark Core
• Assignee: Marcelo Masiero Vanzin
• Reporter: Marcelo Masiero Vanzin
• Created: 15/Nov/17 21:33
• Updated: 20/Nov/17 11:46
• Resolved: 20/Nov/17 11:45

Description

Noticed that when you read a config using a ConfigBuilder-created constant using SparkConf.get, that does not respect the deprecated keys declared in SparkConf.configsWithAlternatives.

For example, adding this test to SparkConfSuite generates a failure:

    conf.set("spark.history.fs.cleaner.interval.seconds", "42")
    assert(conf.get(MAX_LOG_AGE_S) === 42L)

While here it would also be nice to handle this TODO from SparkConf.configsWithAlternatives:

   * TODO: consolidate it with `ConfigBuilder.withAlternative`.

Issue Links

• links to: [Github] Pull Request #19760 (vanzin)