Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39592

Asynchronous State Checkpointing in Structured Streaming

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.3.0
    • None
    • Structured Streaming
    • None

    Description

      We can reduce the latency of stateful pipelines in Structured Streaming by making state checkpoints asynchronous.  One of the major contributors of latency for stateful pipelines in Structured Streaming can be checkpointing the state changes of every micro-batch.  If we make the state checkpointing asynchronous, we can potentially significantly lower the latency of the pipeline as the state checkpointing won’t or will contribute less to the batch latency.

      Attachments

        Issue Links

          Is contained by

          Umbrella - An overarching type made of sub-tasks SPARK-40025 Project Lightspeed: Faster and Simpler Stream Processing with Apache Spark

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              jerrypeng Boyang Jerry Peng
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated: