[TAJO-501] Rewrite the projection part of logical planning

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: Planner/Optimizer
• Assignee: Hyunsik Choi
• Reporter: Hyunsik Choi
• Created: 14/Jan/14 09:56
• Updated: 20/Jan/14 01:50
• Resolved: 20/Jan/14 01:50

Description

The projection part of LogicalPlanner was designed long time ago. It has evolved to support many SQL expressions. However, due to its rough design, it is hard to be improved for further SQL expressions and it causes many bugs.

The current logical planner has the following problems:

• other expressions except for column can be used in group-by clause.
  • TAJO-422
• other expressions except for column can not be used in order-by clause.
  • TAJO-444
• An expression including some aggregation function must be evaluated in group-by executor.
  • As a result, some aggregation operator like HashAggregateExec has to keep all intermediate results of a complex expression in a hash table.
  • It also causes frequent GC and large memory consumption.

The too high code complexity also causes many bugs like

• TAJO-434 - java.lang.NullPointerException for invalid column name
• TAJO-428 - CASE WHEN IS NULL condition is a problem using LEFT OUTER JOIN
• TAJO-463 - ProjectionPushDownRule incorrectly rewrite the output schema of StoreTableNode
• TAJO-443 - Order by query gives NullPointerException at at org.apache.tajo.catalog.Schema.getColumnId(Schema.java:142)

The major reason of this problem is as follows:

• TargetListManager keeps only the final target list.
  • SELECT col1, sum(col2) as col2, ... <- the final target list
• TargetListManager deals with each expression described in a target list or other clauses like group-by clause as a singleton expression.

The main objective of this issue is to rewrite the projection part of logical planning in order to those problems.

For 2 weeks, I've rewritten this part. I'll submit the patch soon.