[SPARK-36696] spark.read.parquet loads empty dataset

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: SQL
• Target Version/s: 3.2.0
• Assignee: Unassigned
• Reporter: Takuya Ueshin
• Created: 08/Sep/21 18:50
• Updated: 15/Sep/21 19:51
• Resolved: 15/Sep/21 19:51

Description

Here's a parquet file Spark 3.2/master can't read properly.

The file was stored by pandas and must contain 3650 rows, but Spark 3.2/master returns an empty dataset.

>>> import pandas as pd
>>> len(pd.read_parquet('/path/to/example.parquet'))
3650

>>> spark.read.parquet('/path/to/example.parquet').count()
0

I guess it's caused by the parquet 1.12.0.

When I reverted two commits related to the parquet 1.12.0 from branch-3.2:

• https://github.com/apache/spark/commit/e40fce919ab77f5faeb0bbd34dc86c56c04adbaa
• https://github.com/apache/spark/commit/cbffc12f90e45d33e651e38cf886d7ab4bcf96da

it reads the data successfully.

We need to add some workaround, or revert the commits.

Issue Links

• is related to: PARQUET-2078 Failed to read parquet file after writing with the same parquet version
• is related to: SPARK-34276 Check the unreleased/unresolved JIRAs/PRs of Parquet 1.11 and 1.12
• relates to: SPARK-36726 Upgrade Parquet to 1.12.1