Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12795

Whole stage codegen

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Epic
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.0.0
    • SQL
    • None
    • Whole stage codegen

    Description

      Whole stage codegen is used by some modern MPP databases to archive great performance. See http://www.vldb.org/pvldb/vol4/p539-neumann.pdf

      For Spark SQL, we can compile multiple operator into a single Java function to avoid the overhead from materialize rows and Scala iterator.

      Attachments

         
        		 SPARK-12796 initial prototype: projection/filter/range New Feature Resolved Davies Liu
        Actions
         
        		 SPARK-12797 Aggregation without grouping keys New Feature Resolved Davies Liu
        Actions
         
        		 SPARK-12798 Broadcast hash join New Feature Resolved Davies Liu
        Actions
         
        		 SPARK-12902 Visualization and metrics for generated operators Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-12913 Reimplement stat functions as declarative function Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-12914 Generate TungstenAggregate with grouping keys Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-12915 SQL metrics for generated operators Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-12949 Support common expression elimination Improvement Resolved L. C. Hsieh
        Actions
         
        		 SPARK-12950 Improve performance of BytesToBytesMap Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-12951 Support spilling in generate aggregate Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-13031 Improve test coverage for whole stage codegen Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-13095 improve performance of hash join with dimension table Bug Resolved Davies Liu
        Actions
         
        		 SPARK-13123 Add wholestage codegen for sort Bug Resolved Sameer Agarwal
        Actions
         
        		 SPARK-13130 Make whole stage codegen variable names slightly easier to read Improvement Resolved Reynold Xin
        Actions
         
        		 SPARK-13135 Don't print expressions recursively in generated code Improvement Resolved Dongjoon Hyun
        Actions
         
        		 SPARK-13147 improve readability of generated code Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-13237 Generate broadcast outer join Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-13293 Generate code for Expand Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-13304 Broadcast join with two ints could be very slow Bug Resolved Davies Liu
        Actions
         
        		 SPARK-13373 Generate code for sort merge join New Feature Resolved Davies Liu
        Actions
         
        		 SPARK-13404 Create the variables for input when it's used Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-13873 Avoid the copy in whole stage codegen when there is no joins Improvement Resolved Davies Liu
        Actions
         
        		 SPARK-13917 Generate code for broadcast left semi join New Feature Resolved Davies Liu
        Actions
         
        		 SPARK-13950 Generate code for sort merge left/right outer join New Feature Closed Davies Liu
        Actions
         
        		 SPARK-14710 Rename gen/genCode to genCode/doGenCode to better reflect the semantics Improvement Resolved Sameer Agarwal
        Actions
         
        		 SPARK-14718 Avoid mutating ExprCode in doGenCode Improvement Resolved Sameer Agarwal
        Actions
         
        		 SPARK-14722 Rename upstreams() -> inputRDDs() in WholeStageCodegen Improvement Resolved Sameer Agarwal
        Actions
         
        		 SPARK-14748 BoundReference should not set ExprCode.code to empty string Improvement Closed Unassigned
        Actions
         
        		 SPARK-16844 Generate code for sort based aggregation New Feature Resolved Unassigned
        Actions

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            davies Davies Liu
            davies Davies Liu
            Votes:
            0 Vote for this issue
            Watchers:
            34 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment