[DRILL-1115] parquet writer never finishes when the source contains huge number of small files

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Component/s: Storage - Writer
• Assignee: Rahul Kumar Challapalli
• Reporter: Rahul Kumar Challapalli
• Due: 15/Aug/14
• Created: 08/Jul/14 00:23
• Updated: 04/Jan/15 21:28
• Resolved: 04/Jan/15 21:28

Description

git.commit.id.abbrev=790a2ad
Build # 26246

When we try to use 'create table as.....' and the source folder contains around 5000 text files, drill never completes. I left the query to tun overnight, but it still didn't complete. However I see nonstop activity in the log files which suggests drill is actually doing something.

Cluster Size : 2
Each file contains only a single number.