[DRILL-6101] Optimize Implicit Columns Processing

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Done
• Affects Version/s: 1.12.0
• Fix Version/s: 1.15.0
• Component/s: Execution - Relational Operators
• Labels: ready-to-commit
• Assignee: Salim Achouche
• Reporter: Salim Achouche
• Reviewer: Timothy Farkas
• Created: 22/Jan/18 19:13
• Updated: 03/Aug/18 16:46
• Resolved: 03/Aug/18 16:46

Description

Problem Description -

• Apache Drill allows users to specify columns even for SELECT STAR queries
• From my discussion with paul-rogers, Apache Calcite has a limitation where the, extra columns are not provided
• The workaround has been to always include all implicit columns for SELECT STAR queries
• Unfortunately, the current implementation is very inefficient as implicit column values get duplicated; this leads to substantial performance degradation when the number of rows are large

Suggested Optimization -

• The NullableVarChar vector should be enhanced to efficiently store duplicate values
• This will not only address the current Calcite limitations (for SELECT STAR queries) but also optimize all queries with implicit columns

 

Issue Links

• links to: GitHub Pull Request #1414