Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13404

Create the variables for input when it's used

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Right now, we create the variables in the first operator (usually InputAdapter), they could be wasted if most of rows after filtered out immediately.

      We should defer that until they are used by following operators.

        Issue Links

          Activity

          Hide
          apachespark Apache Spark added a comment -

          User 'davies' has created a pull request for this issue:
          https://github.com/apache/spark/pull/11274

          Show
          apachespark Apache Spark added a comment - User 'davies' has created a pull request for this issue: https://github.com/apache/spark/pull/11274
          Hide
          davies Davies Liu added a comment -

          Issue resolved by pull request 11274
          https://github.com/apache/spark/pull/11274

          Show
          davies Davies Liu added a comment - Issue resolved by pull request 11274 https://github.com/apache/spark/pull/11274

            People

            • Assignee:
              davies Davies Liu
              Reporter:
              davies Davies Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development