[SPARK-45334] Remove misleading comment in parquetSchemaConverter

• Type: Documentation
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 3.5.0
• Fix Version/s: 4.0.0
• Component/s: SQL
• Labels: pull-request-available
• Assignee: Mengran Lan
• Reporter: Mengran Lan
• Shepherd: Hyukjin Kwon
• Created: 26/Sep/23 06:40
• Updated: 27/Sep/23 02:01
• Resolved: 27/Sep/23 02:01

Description

I'm debugging a parquet issue and reading spark code as references. Happened to find a misleading comment which remains in the latest version as well.

Types
  .buildGroup(repetition).as(LogicalTypeAnnotation.listType())
  .addField(Types
    .buildGroup(REPEATED)
    // "array" is the name chosen by parquet-hive (1.7.0 and prior version)
    .addField(convertField(StructField("array", elementType, nullable)))
    .named("bag"))
  .named(field.name) 

the comment above is misleading since Hive always uses "array_element" as the name.

It is imported by this PR https://github.com/apache/spark/pull/14399 and relates to this issue https://issues.apache.org/jira/browse/SPARK-16777

Furthermore, the parquet-hive module has been removed from the parquet-mr project https://issues.apache.org/jira/browse/PARQUET-1676 

I suggest removing this piece of comment and will submit a PR later.

Issue Links

• links to: GitHub Pull Request #43119