Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33659

Document the current behavior for DataStreamWriter.toTable API

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 3.1.0
    • 3.1.0
    • Structured Streaming
    • None

    Description

      Follow up work for SPARK-33577 and need to be done before the 3.1 release. As we didn't have full support for the V2 table created in the API, the following documentation work is needed:

      • figure out the effects when configurations are (provider/partitionBy) conflicting with existing table, and document in javadoc of toTable. I think you'll need to make a matrix and describe which takes effect (table vs input) - creating table vs table exists, DSv1 vs DSv2 (4 different situations should be all documented).
      • document the lack of functionality on creating v2 table in javadoc of toTable, and guide that they should ensure table is created in prior to avoid the behavior unintended/insufficient table is being created.

      Attachments

        Activity

          People

            XuanYuan Yuanjian Li
            XuanYuan Yuanjian Li
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: