Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3196

Expression Evaluation Performance Improvement

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      The expression id generations depend on a atomic long object internally, which will cause the performance drop dramatically in a multi-threading execution.

      I'd like to create 2 sub tasks(maybe more) for the improvements:

      1) Reduce the expression tree object creation from the aggregation functions (min/max), as they will create expression trees for each single row.
      2) Remove the id property from the TreeNode API.

      And remove the expression object creation as many as possible, where we have the expression evaluation. (I will create couple of subtask soon).

        Attachments

        There are no Sub-Tasks for this issue.

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              chenghao Cheng Hao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: