[DRILL-285] Some operators fail to process all records

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Assignee: Jacques Nadeau
• Reporter: Ben Becker
• Created: 08/Nov/13 21:05
• Updated: 31/Jul/14 06:32
• Resolved: 29/May/14 15:44

Description

The test below shows that the Filter operator cannot process more than more than 15625 records from the mock scanner. The issue may affect multiple operators and scanners.

Steps to reproduce:

1. Edit filter/test1.json and increase the record count to some number above 15625 (e.g. 20k).
2. Update the assertion in TestSimpleFilter.testFilter().
3. Run TestSimpleFilter.testFilter().

Expected Results:

The test would return 1/2 the record count.

Results:

FilterRecordBatch.doWork() gets a value of 15625 from incoming.getRecordCount(). Only one batch is processed, and never more than 15625 records. The same holds true when the mock reader generates multiple batches.