[SPARK-32550] Make SpecificInternalRow constructors faster by using while loops instead of maps

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.1.0
• Component/s: SQL
• Assignee: Muhammad Samir Khan
• Reporter: Muhammad Samir Khan
• Created: 05/Aug/20 23:23
• Updated: 12/Dec/22 18:10
• Resolved: 19/Aug/20 05:58

Description

Two constructors in SpecificInternalRow can be made faster by using while loops instead of maps. This was originally noticed while working on SPARK-32532 and SPARK-32533 and will have impacts on performance of reading ORC and Avro files. Profiled the change using added benchmarks in SPARK-32531 for nested and array'd structs.

Issue Links

• is duplicated by: SPARK-32690 Spark-32550 affects the performance of some cases
• is related to: SPARK-32532 Improve ORC read/write performance on nested structs and array of structs
• is related to: SPARK-32533 Improve Avro read/write performance on nested structs and array of structs
• is related to: SPARK-32531 Add benchmarks for nested structs and arrays for different file formats
• links to: [Github] Pull Request #29366 (msamirkhan)
• links to: [Github] Pull Request #29366 (msamirkhan)
• links to: [Github] Pull Request #29529 (LuciferYang)
• links to: [Github] Pull Request #29529 (LuciferYang)