[DRILL-6853] Parquet Complex Reader for nested schema should have configurable memory or max records to fetch

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.15.0
• Labels: doc-complete pull-request-available ready-to-commit
• Assignee: Salim Achouche
• Reporter: Nitin Sharma
• Reviewer: Timothy Farkas
• Created: 15/Nov/18 06:51
• Updated: 07/Jan/19 22:58
• Resolved: 16/Nov/18 00:40

Description

Parquet Complex reader while fetching nested schema should have configurable memory or max records to fetch and not default to 4000 records.

While scanning TB of data with wider columns, this could easily cause OOM issues. 

Issue Links

• links to: GitHub Pull Request #1544