Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.1.0
    • 3.3.0
    • Optimizer
    • None

    Description

      I propose to add tree pattern pruning to the CTESubstitution rule in order to skip tree traversal when the tree does not contain an UnresolvedWith expression.

      This is motivated by profiling a job which uses DataFrame APIs to incrementally construct a huge query plan (200k+ nodes): each API call results in eager re-analysis of the plan, of which CTESubstitution accounts for the majority of the analysis time. This query didn't contain CTEs, so skipping the CTESubstitution significantly speeds up analysis.

      I plan to submit a patch for this.

      Attachments

        Activity

          People

            joshrosen Josh Rosen
            joshrosen Josh Rosen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: