[TAJO-104] JIT Query Compilation and Vectorized Engine (Umbrella)

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: Physical Operator, Worker
• Labels: vectorization
• Assignee: Unassigned
• Reporter: Hyunsik Choi
• Created: 03/Aug/13 07:54
• Updated: 15/Oct/15 04:40
• Resolved: 15/Oct/15 04:40

Description

In these days, it's unnecessary to say the advantages of columnar store and vectorized processing on analytic workloads. These approaches are well known as the state-of-the-art techniques in database community and are also acceptable in practical areas.

Since we started Tajo project in 2010 year, we have planed the new engine using both JIT query compilation and vectorized engine. My colleagues and I have surveyed columnar store, vectorized processing, cache conscious techniques, and query compilation.

In this issue, we will design and implement the new engine. The key implementation plan is as follows:

• Implemented in C++
• Vectorization primitives will be generated by LLVM.
• Two or more primitives by using JIT can be blurred according to the situation.

This is an umbrella issue, and we will create lots of subtasks for this issue.

The design references are as follows:

• DSM vs. NSM: CPU Performance Tradeoffs in Block-Oriented Query Processing.
• Efficiently Compiling Efficient Query Plans for Modern Hardware
• Just-in-time Compilation in Vectorized Query Execution
• MonetDB/X100: Hyper-Pipelining Query Execution
• Column-Stores vs. Row-Stores: How Different Are They Really?
• Balancing vectorized query execution with bandwidth-optimized storage

Sub-Tasks

• the initial code layout for new engine: Resolved
• Design and Implement Vectors class: Resolved
• Implement column-column filter primitive generator: Resolved
• Implement column-value filter primitive generator: Resolved
• Implement column-scalar expression primitive generator: Resolved
• Implement column-column expression primitive generator: Resolved
• Implement IS NULL filter primitive function: Resolved
• Implement LIKE filter primitive function: Resolved
• Implement MIN aggregation primitive generator: Resolved
• Implement MAX aggregation primitive generator: Resolved
• Implement SUM aggregation primitive generator: Resolved
• Implement count rows (i.e., count(*) ) aggregation primitive function: Resolved
• Implement count column (i.e., count(col) ) aggregation primitive function: Resolved
• Implement a physical planner to generate an execution plan: Resolved
• Implement C++ RPC module: Resolved
• Integrate with Java TajoMaster: Resolved