Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-38836

Increase the performance of ExpressionSet

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.2.1
    • 3.4.0
    • Optimizer
    • None

    Description

      Since the constructor of `ExpressionSet` is protected, callers can only modify the state of an `ExpressionSet` by calling `add` and `remove`, so it maintains the invariants of:

      1. Every expr `e` in `baseSet` satisfies `e.deterministic && e.canonicalized == e`
      2. Every deterministic expr `e` in `originals` satisfies that `e.canonicalized` is already accessed.

      The current implementation ignores the invariant and unnecessarily calls `.canonicalized` in lots of places which can cause performance issues as `canonicalized` is a pretty expensive operation.

      Attachments

        Activity

          People

            miny Min Yang
            miny Min Yang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: