Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33183

Bug in optimizer rule EliminateSorts

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 3.0.2, 3.1.0
    • Fix Version/s: 2.4.8, 3.0.2, 3.1.0
    • Component/s: SQL
    • Labels:

      Description

      Currently, the rule EliminateSorts removes a global sort node if its child plan already satisfies the required sort order without checking if the child plan's ordering is local or global. For example, in the following scenario, the first sort shouldn't be removed because it has a stronger guarantee than the second sort even if the sort orders are the same for both sorts. 

      Sort(orders, global = True, ...)
        Sort(orders, global = False, ...)

       

        Attachments

          Activity

            People

            • Assignee:
              allisonwang-db Allison Wang
              Reporter:
              allisonwang-db Allison Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: