[SPARK-37980] Extend METADATA column to support row indices for file based data sources

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.4.0
• Component/s: SQL
• Assignee: Ala Luszczak
• Reporter: Prakhar Jain
• Created: 21/Jan/22 16:13
• Updated: 22/Feb/23 12:32
• Resolved: 16/Aug/22 13:04

Description

Spark recently added hidden metadata column support for File based datasources as part of  SPARK-37273.

We should extend it to support ROW_INDEX/ROW_POSITION also.

 

Meaning of  ROW_POSITION:

ROW_INDEX/ROW_POSITION is basically an index of a row within a file. E.g. 5th row in the file will have ROW_INDEX 5.

 

Use cases: 

Row Indexes can be used in a variety of ways. A (fileName, rowIndex) tuple uniquely identifies row in a table. This information can be used to mark rows e.g. this can be used by indexer etc.

Issue Links

• causes: SPARK-39634 Allow file splitting in combination with row index generation
• is blocked by: SPARK-39806 Queries accessing METADATA struct crash on partitioned tables
• relates to: PARQUET-2117 Add rowPosition API in parquet record readers
• relates to: SPARK-40059 Row indexes can overshadow user-created data
• links to: [Github] Pull Request #37228 (ala)
• links to: [Github] Pull Request #40124 (olaky)