[PARQUET-98] filter2 API performance regression

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Viktor Szathmáry
• Created: 22/Sep/14 22:24
• Updated: 23/Jun/24 03:26

Description

The new filter API seems to be much slower (or perhaps I'm using it wrong :)

Code using an UnboundRecordFilter:

ColumnRecordFilter.column(column,
    ColumnPredicates.applyFunctionToBinary(
    input -> Binary.fromString(value).equals(input)));

vs. code using FilterPredicate:

eq(binaryColumn(column), Binary.fromString(value));

The latter performs twice as slow on the same Parquet file (built using 1.6.0rc2).

Note: the reader is constructed using

ParquetReader.builder(new ProtoReadSupport().withFilter(filter).build()

The new filter API based approach seems to create a whole lot more garbage (perhaps due to reconstructing all the rows?).

Issue Links

• is related to: PARQUET-182 FilteredRecordReader skips rows it shouldn't for schema with optional columns