Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41261

applyInPandasWithState can produce incorrect key value in user function for timed out state

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Structured Streaming
    • None

    Description

      We observed the issue that user function retrieves incorrect key in user function for timed out state. After RCA we figured out this could happen when the columns of grouping keys are not placed sequentially at earliest place.

      Attachments

        Activity

          People

            kabhwan Jungtaek Lim
            kabhwan Jungtaek Lim
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: