Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24761

Check modifiability of config parameters

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.1
    • Fix Version/s: 2.4.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Our customers and support team continuously face to the situation when setting a config parameter via spark.conf.set() does not may any effects. It is not clear from parameter's name is it static parameter or one of the parameter that can be set at runtime for current session state. It would be useful to have a method of RuntimeConfig which could tell to an user - does the given parameter may effect on the current behavior if he/she change it in the spark-shell or running notebook. The method can have the following signature:

      def isModifiable(key: String): Boolean
      

      Any config parameter can be checked by using the syntax like this:

      scala> spark.conf.isModifiable("spark.sql.sources.schemaStringLengthThreshold")
      res0: Boolean = false
      

      or for Spark Core parameter:

      scala> spark.conf.isModifiable("spark.task.cpus")
      res1: Boolean = false
      

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              maxgekk Max Gekk
              Reporter:
              maxgekk Max Gekk

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment