Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22911 Migrate structured streaming sources to new DataSourceV2 APIs
  3. SPARK-23558

clean up StreamWriter factory lifecycle

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.4.0
    • None
    • Structured Streaming

    Description

      Right now, StreamWriter and children have different lifecycles in continuous processing and microbatch mode. Both execution modes impose significant constraints on what that lifecycle must be, so the achievable consistent semantic is:

      • StreamWriter lasts for the duration of the query execution
      • DataWriterFactory lasts for the duration of the query execution
      • DataWriter (the task-level writer) has a lifecycle tied to each individual epoch

      This also allows us to restore the implicit semantic that DataWriter.commit()/abort() terminates the lifecycle.

      Attachments

        Activity

          People

            Unassigned Unassigned
            joseph.torres Jose Torres
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: