[DRILL-8261] Make the XML format plugin's attributes MAP an implicit column

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Not A Problem
• Affects Version/s: 1.20.1
• Fix Version/s: 1.21.0
• Component/s: Storage - XML
• Assignee: James Turton
• Reporter: James Turton
• Created: 19/Jul/22 16:07
• Updated: 26/Jul/22 01:32
• Resolved: 26/Jul/22 01:32

Description

The attributes MAP column generated by the XML format plugin is currently explicit and present in wildcard selects. However, attributes are frequently not present at all in XML data, or are better queried using explicit projections of the individual attributes of interest to scalars. The motivating example here is an ETL-style query that transforms XML to Parquet using a CTAS with a wildcard column spec. This query will fail for XML that has no attributes because the Parquet writer cannot write a Parquet schema containing the empty struct produced by the attributes map. It is therefore proposed that the attributes MAP becomes an implicit column.

Issue Links

• is fixed by: DRILL-8272 Skip MAP column without children when creating parquet tables
• is related to: SPARK-20593 Writing Parquet: Cannot build an empty group