Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-501

Rewrite the projection part of logical planning

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Planner/Optimizer
    • Labels:
      None

      Description

      The projection part of LogicalPlanner was designed long time ago. It has evolved to support many SQL expressions. However, due to its rough design, it is hard to be improved for further SQL expressions and it causes many bugs.

      The current logical planner has the following problems:

      • other expressions except for column can be used in group-by clause.
      • other expressions except for column can not be used in order-by clause.
      • An expression including some aggregation function must be evaluated in group-by executor.
        • As a result, some aggregation operator like HashAggregateExec has to keep all intermediate results of a complex expression in a hash table.
        • It also causes frequent GC and large memory consumption.

      The too high code complexity also causes many bugs like

      • TAJO-434 - java.lang.NullPointerException for invalid column name
      • TAJO-428 - CASE WHEN IS NULL condition is a problem using LEFT OUTER JOIN
      • TAJO-463 - ProjectionPushDownRule incorrectly rewrite the output schema of StoreTableNode
      • TAJO-443 - Order by query gives NullPointerException at at org.apache.tajo.catalog.Schema.getColumnId(Schema.java:142)

      The major reason of this problem is as follows:

      • TargetListManager keeps only the final target list.
        • SELECT col1, sum(col2) as col2, ... <- the final target list
      • TargetListManager deals with each expression described in a target list or other clauses like group-by clause as a singleton expression.

      The main objective of this issue is to rewrite the projection part of logical planning in order to those problems.

      For 2 weeks, I've rewritten this part. I'll submit the patch soon.

        Attachments

        1. TAJO-501_4.patch
          610 kB
          Hyunsik Choi
        2. TAJO-501_3.patch
          609 kB
          Hyunsik Choi
        3. TAJO-501_2.patch
          609 kB
          Hyunsik Choi
        4. TAJO-501.patch
          609 kB
          Hyunsik Choi

          Activity

            People

            • Assignee:
              hyunsik Hyunsik Choi
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: