Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39551

Add AQE invalid plan check

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete CommentsDelete
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.3.0
    • 3.2.2, 3.3.1, 3.4.0
    • SQL
    • None

    Description

      AQE logical optimization rules can lead to invalid physical plans as certain physical plan nodes are not compatible with others. E.g., `BroadcastExchangeExec` can only work as a direct child of broadcast join nodes.

      Logical optimizations, on the other hand, are not (and should not be) aware of such restrictions. So a general solution here is to check for invalid plans and throw exceptions, which can be caught by AQE replanning process. And if such an exception is captured, AQE can void the current replanning result and keep using the latest valid plan.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            maryannxue Wei Xue Assign to me
            maryannxue Wei Xue
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment