Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33183

Bug in optimizer rule EliminateSorts

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 2.4.5, 2.4.6, 2.4.7, 2.4.8, 3.0.2, 3.1.0
    • 2.4.8, 3.0.2, 3.1.0
    • SQL

    Description

      Currently, the rule EliminateSorts removes a global sort node if its child plan already satisfies the required sort order without checking if the child plan's ordering is local or global. For example, in the following scenario, the first sort shouldn't be removed because it has a stronger guarantee than the second sort even if the sort orders are the same for both sorts. 

      Sort(orders, global = True, ...)
        Sort(orders, global = False, ...)

       

      Attachments

        Activity

          People

            allisonwang-db Allison Wang
            allisonwang-db Allison Wang
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: