[DRILL-6147] Limit batch size for Flat Parquet Reader

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Done
• Fix Version/s: 1.14.0
• Component/s: Storage - Parquet
• Labels: ready-to-commit
• Assignee: Salim Achouche
• Reporter: Salim Achouche
• Reviewer: Timothy Farkas
• Created: 09/Feb/18 17:35
• Updated: 30/Jun/18 01:54
• Resolved: 30/Jun/18 01:54

Description

The Parquet reader currently uses a hard-coded batch size limit (32k rows) when creating scan batches; there is no parameter nor any logic for controlling the amount of memory used. This enhancement will allow Drill to take an extra input parameter to control direct memory usage.

Issue Links

• links to: GitHub Pull Request #1294
• links to: GitHub Pull Request #1330
• links to: Parquet batch sizing functionality