Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-42163

Schema pruning fails on non-foldable array index or map key

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.2.3
    • 3.4.0
    • Optimizer
    • None

    Description

      Schema pruning tries to extract selected fields from struct extractors. It looks through GetArrayItem/GetMapItem, but when doing so, it ignores the index/key, which may itself be a struct field. If it is a struct field that is not otherwise selected, and some other field of the same attribute is selected, then pruning will drop the field, resulting in an optimizer error.

      Attachments

        Activity

          People

            David Cashman David Cashman
            David Cashman David Cashman
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: