Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22386 Data Source V2 improvements
  3. SPARK-28555

Recover options and properties and pass them back into the v1 API

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      When tables are created, the CREATE TABLE syntax supports both TBLPROPERTIES and OPTIONS. Options were used in v1 to configure the table itself, like options passed to DataFrameReader. Right now, both properties and options are stored in v2 table properties, because v2 only has properties, not both. But, we aren't able to recover which properties were set through OPTIONS and which were set through TBLPROPERTIES.

      Instead of the current behavior, I think options should be prefixed with option.. That way, we can recover options and properties and pass them back into the v1 API.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              iamshrek Xin Ren
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: