Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33962

Fix incorrect min partition condition in getRanges

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • Structured Streaming
    • None

    Description

      When calculating offset ranges, we consider minPartitions configuration. If minPartitions is not set or is less than or equal the size of given ranges, it means there are enough partitions at Kafka so we don't need to split offsets to satisfy min partition requirement. But the current condition is offsetRanges.size > minPartitions.get and is not correct. Currently getRanges will split offsets in unnecessary case.

      Attachments

        Activity

          People

            viirya L. C. Hsieh
            viirya L. C. Hsieh
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: