Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-41539

stats and constraints in LogicalRDD may not be in sync with output attributes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • SQL
    • None

    Description

      We encountered the case where the output of logical plan and optimized plan were different in LogicalRDD (the difference was exprId for the case), led the situation that stats and constraints are out of sync with output attributes, eventually failed the query.

      We should remap stats and constraints based on the output of logical plan, assuming that the output of logical plan and optimized plan are "slightly" different (e.g. exprId) but "semantically" same.

      Attachments

        Activity

          People

            kabhwan Jungtaek Lim
            kabhwan Jungtaek Lim
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: