Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-10316

respect non-deterministic expressions in PhysicalOperation

          Details

          • Type: Bug
          • Status: Resolved
          • Priority: Major
          • Resolution: Fixed
          • Affects Version/s: None
          • Fix Version/s: 1.6.0
          • Component/s: SQL
          • Labels:
            None

            Description

            We did a lot of special handling for non-deterministic expressions in Optimizer. However, PhysicalOperation just collects all Projects and Filters and messed it up. We should respect the operators order caused by non-deterministic expressions in PhysicalOperation.

              Attachments

                Issue Links

                causes

                Bug - A problem which impairs or prevents the functions of the product. SPARK-27969 Non-deterministic expressions in filters or projects can unnecessarily prevent all scan-time column pruning, harming performance

                • Major - Major loss of function.
                • Open
                links to

                Pull request #8486 [Github] Pull Request #8486 (cloud-fan)

                Web Link GitHub Pull Request #8486

                  Activity

                    People

                    • Assignee:
                      cloud_fan Wenchen Fan
                      Reporter:
                      cloud_fan Wenchen Fan
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      3 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: