[PARQUET-4] Use LRU caching for footers in ParquetInputFormat.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: parquet-mr
• Assignee: Matt Martin
• Reporter: Matt Martin
• Created: 20/Jun/14 03:45
• Updated: 23/Jun/24 03:23
• Resolved: 18/Jul/14 23:02

Description

The caching approach needs to change because of issues that occur when the same ParquetInputFormat instance is reused to generate splits for different input directories. For example, it causes problems in Hive's FetchOperator when the FetchOperator is attempting to operate over more than one partition (sidenote: as far as I could tell, Hive has been reusing inputformat instances in this way for quite some time). The details of how this issue manifests itself with respect to Hive are described in more detail here: https://groups.google.com/d/msg/parquet-dev/0aXql-3z7vE/Gn5m094V7PMJ

The proposed patch can be found here: https://github.com/apache/incubator-parquet-mr/pull/2

Issue Links

• is related to: PARQUET-16 Unnecessary getFileStatus() calls on all part-files in ParquetInputFormat.getSplits