Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22533

SparkConfigProvider does not handle deprecated config keys

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      Noticed that when you read a config using a ConfigBuilder-created constant using SparkConf.get, that does not respect the deprecated keys declared in SparkConf.configsWithAlternatives.

      For example, adding this test to SparkConfSuite generates a failure:

          conf.set("spark.history.fs.cleaner.interval.seconds", "42")
          assert(conf.get(MAX_LOG_AGE_S) === 42L)
      

      While here it would also be nice to handle this TODO from SparkConf.configsWithAlternatives:

         * TODO: consolidate it with `ConfigBuilder.withAlternative`.
      

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Vanzin
              Reporter:
              vanzin Marcelo Vanzin
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: