Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15689 Data source API v2
  3. SPARK-22387

propagate session configs to data source read/write options

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.3.0
    • 2.3.0
    • SQL
    • None

    Description

      This is an open discussion. The general idea is we should allow users to set some common configs in session conf so that they don't need to type them again and again for each data source operations.

      Proposal 1:
      propagate every session config which starts with spark.datasource.config. to data source options. The downside is, users may only want to set some common configs for a specific data source.

      Proposal 2:
      propagate session config which starts with spark.datasource.config.myDataSource. only to myDataSource operations. One downside is, some data source may not have a short name and makes the config key pretty long, e.g. spark.datasource.config.com.company.foo.bar.key1.

      Proposal 3:
      Introduce a trait `WithSessionConfig` which defines session config key prefix. Then we can pick session configs with this key-prefix and propagate it to this particular data source.

      One another thing also worth to think: sometimes it's really annoying if users have a typo in the config key and spend a lot of time to figure out why things don't work as expected. We should allow data source to validate the given options and throw exception if an option can't be recognized.

      Attachments

        Activity

          People

            jiangxb1987 Xingbo Jiang
            cloud_fan Wenchen Fan
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: