Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17924

Consolidate streaming and batch write path

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      Structured streaming and normal SQL operation currently have two separate write path, leading to a lot of duplicated functions (that look similar) and if branches. The purpose of this ticket is to consolidate the two as much as possible to make the write path more clear.

      A side-effect of this change is that streaming will automatically support all the file formats.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rxin Reynold Xin
                Reporter:
                rxin Reynold Xin
              • Votes:
                1 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: