[PARQUET-1602] PageIndex not working as suggested ?

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.11.0
• Component/s: parquet-mr
• Assignee: Unassigned
• Reporter: Anthony Pessy
• Created: 18/Jun/19 10:18
• Updated: 18/Jun/19 14:26
• Resolved: 18/Jun/19 14:26

Description

I have a schema such as:

schema message pages {
  required binary url (STRING);
  optional binary content (STRING);
}

Where `url` is unique and ordered, the file is created in such a way that I have ~600 pages of `content` for 1 page of `url`.

 

From https://github.com/apache/parquet-format/blob/master/PageIndex.md I saw:

A single-row lookup in a rowgroup based on the sort column of that rowgroup will only read one data page per retrieved column.

I was expecting `ParquetReader`  to find the matching row thanks to the `FilterPredicate`  on `url`, decoding only this column, then, using `offset index`, directly seek to the appropriate page for `content` and decode it.

Instead, what I'm seeing, is that the reader fully reads & decode the ~600 pages of content (until it actually find the url).

Is there something I misunderstood or some step to ensure to make the reader only consume the necessary pages?