[DRILL-1095] Add support for large number of columns in queries

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Component/s: Execution - Codegen, Metadata, Storage - Writer
• Assignee: Aditya Kishore
• Reporter: Rahul Kumar Challapalli
• Created: 02/Jul/14 00:29
• Updated: 31/Jul/14 05:53
• Resolved: 31/Jul/14 05:53

Description

git.commit.id.abbrev=a1a6144
Build# 26246

The 'create table as' statement starts to break when the column count is somewhere around 125-150. I attached the error log. Let me know if you have any questions.

The root cause of this (and similar failures) is that when a Drill operator needs to process (evaluate/copy/spool) a large number of expressions (columns), the resultant runtime generated code can exceed the 64 KB byte code limit imposed on the size of a function by Java class file specifications, resulting in compilation failure.

This umbrella JIRA will serve as a container for various sub-tasks to address this issue.

Sub-Tasks

• Add option to use JDK compiler and use it for runtime generated code over a certain threshold: Resolved
• Session option to enable/disable debug information in runtime generated Java code: Resolved
• Split large runtime generated functions into a chain of functions: Resolved