Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30761

Nested pruning should not prune on required child outputs in Generate

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Cannot Reproduce
    • 3.0.0
    • None
    • SQL
    • None

    Description

      We prune nested fields from Generate. If a child output is required in a top operator of Generate, we should not prune nested fields on it. Otherwise, the accessors on top operator could be unresolved.

      Attachments

        Issue Links

          is caused by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-29721 Spark SQL reads unnecessary nested fields after using explode

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #27503

          Activity

            People

              Unassigned Unassigned
              viirya L. C. Hsieh
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: