Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15406 Structured streaming support for consuming from Kafka
  3. SPARK-17813

Maximum data per trigger

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.2, 2.1.0
    • Structured Streaming
    • None

    Description

      At any given point in a streaming query execution, we process all available data. This maximizes throughput at the cost of latency. We should add something similar to the maxFilesPerTrigger option available for files.

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-26718 Fixed integer overflow in SS kafka rateLimit calculation

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #15527 [Github] Pull Request #15527 (koeninger)

          Activity

            People

              koeninger Cody Koeninger
              marmbrus Michael Armbrust
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: