[PARQUET-548] Add Java metadata for PageEncodingStats

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.9.0, 1.8.2
• Component/s: parquet-mr
• Assignee: Ryan Blue
• Reporter: Ryan Blue
• Created: 26/Feb/16 17:41
• Updated: 23/Jun/24 03:28
• Resolved: 23/Apr/16 00:40

Description

PARQUET-384 needs to determine whether an entire column chunk is dictionary-encoded, but it is difficult to detect that case based on the set of encodings for a column. For 1.0, this can be done by checking for a PLAIN page because both dictionary pages and dictionary-encoded pages use PLAIN_DICTIONARY and RLE/BIT_PACKING is only used for repetition and definition levels. But for 2.0, dictionary pages might be using PLAIN and there is no way to tell if a column has fallen back.

PageEncodingStats were added to the format to solve this problem, so we just need to implement them.

Issue Links

• is related to: PARQUET-384 Add Dictionary Based Filtering to Filter2 API
• links to: PR #332