Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32331

Keep advanced statistics when pruning partitions

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 3.1.0
    • None
    • SQL
    • None

    Description

      The rule PruneFileSourcePartitions deletes the advances statistics (cardinality, column stats) of the underlying LogicalRelation and keeps only sizeInBytes as that is the only value we can be sure that is accurate.
      I think we should keep all statistics as they are a good upper limit estimates and by keeping them other rules that depend on the presence of these statistics (like CostBasedJoinReorder) can still ran.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              petertoth Peter Toth
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: