[DRILL-5846] Improve Parquet Reader Performance for Flat Data types

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.14.0
• Component/s: Storage - Parquet
• Labels: performance ready-to-commit
• Assignee: Salim Achouche
• Reporter: Salim Achouche
• Reviewer: Parth Chandra
• Created: 05/Oct/17 20:40
• Updated: 22/Jun/18 18:18
• Resolved: 22/May/18 01:22

Description

The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to further improve the Parquet Reader performance as several users reported that Parquet parsing represents the lion share of the overall query execution. It tracks Flat Data types only as Nested DTs might involve functional and processing enhancements (e.g., a nested column can be seen as a Document; user might want to perform operations scoped at the document level that is no need to span all rows). Another JIRA will be created to handle the nested columns use-case.

Issue Links

• causes: DRILL-6528 Planner setting the wrong number of records to read (Parquet Reader)
• links to: GitHub Pull Request #1060
• links to: GitHub Pull Request #1060

Sub-Tasks

• Flat Parquet Reader Performance Analysis: Resolved
• Implement Parquet Columnar Processing & Use Bulk APIs for processing: Resolved