[SPARK-35500] GenerateSafeProjection.generate will generate SpecificSafeProjection class, but if column is array type or map type, the code cannot be reused which impact the query performance

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.4.5
• Component/s: SQL
• Labels: codegen
• Assignee: Unassigned
• Reporter: Yahui Liu
• Created: 24/May/21 08:15
• Updated: 25/May/21 03:45
• Resolved: 25/May/21 02:45

Description

Reproduce steps:

1. create a new table with array type: create table test_code_gen(a array<int>);
2. Add log4j.logger.org.apache.spark.sql.catalyst.expressions.codegen.CodeGenerator = DEBUG to log4j.properties;
3. Enter spark-shell, fire a query: spark.sql("select * from test_code_gen").collect
4. Everytime, Dataset.collect is called, SpecificSafeProjection class is generated, but the code for the class cannot be reused because everytime the id for two variables in the generated class is changed: MapObjects_loopValue and MapObjects_loopIsNull. So even the class generated before has been cached, new code cannot match the cache key so that new code need to be compiled again which cost some time.  The time cost for compile is increasing with the growth of column number, for wide table, this cost can more than 2s. 

   object MapObjects {
     private val curId = new java.util.concurrent.atomic.AtomicInteger()
    val id = curId.getAndIncrement()
    val loopValue = s"MapObjects_loopValue$id"
    val loopIsNull = if (elementNullable) {
      s"MapObjects_loopIsNull$id"
    } else {
      "false"
    }
   

   First time run: 

   class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
    private int MapObjects_loopValue1;
    private boolean MapObjects_loopIsNull1;
    private UTF8String MapObjects_loopValue2;
    private boolean MapObjects_loopIsNull2;
   }
   

   Second time run:

   class SpecificSafeProjection extends org.apache.spark.sql.catalyst.expressions.codegen.BaseProjection {
    private int MapObjects_loopValue3;
    private boolean MapObjects_loopIsNull3;
    private UTF8String MapObjects_loopValue4;
    private boolean MapObjects_loopIsNull4;
   }

Expectation:
The code generated by GenerateSafeProjection can be reused if the query is same.

Issue Links

• relates to: SPARK-27871 LambdaVariable should use per-query unique IDs instead of globally unique IDs