[SPARK-18860] Update Parquet to 1.9.0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Dongjoon Hyun
• Created: 14/Dec/16 12:53
• Updated: 12/Dec/22 18:10
• Resolved: 14/Dec/16 19:28

Description

This issue aims to update Parquet to 1.9.0 and remove the hacks due to Parquet 1.8.1 limitation.

-  // !! HACK ALERT !!
-  //
-  // PARQUET-363 & PARQUET-278: parquet-mr 1.8.1 doesn't allow constructing empty GroupType,
-  // which prevents us to avoid selecting any columns for queries like `SELECT COUNT(*) FROM t`.
-  // This issue has been fixed in parquet-mr 1.8.2-SNAPSHOT.
-  //
-  // To workaround this problem, here we first construct a `MessageType` with a single dummy
-  // field, and then remove the field to obtain an empty `MessageType`.
-  //
-  // TODO Reverts this change after upgrading parquet-mr to 1.8.2+
   val EMPTY_MESSAGE = Types
       .buildMessage()
-      .required(PrimitiveType.PrimitiveTypeName.INT32).named("dummy")
       .named(ParquetSchemaConverter.SPARK_PARQUET_SCHEMA_NAME)
-  EMPTY_MESSAGE.getFields.clear()

Issue Links

• duplicates: SPARK-13127 Upgrade Parquet to 1.9 (Fixes parquet sorting)
• links to: [Github] Pull Request #16281 (dongjoon-hyun)