[SPARK-32731] Add tests for arrays/maps of nested structs to ReadSchemaSuite to test structs reuse

• Type: Test
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: SQL, Tests
• Assignee: Unassigned
• Reporter: Muhammad Samir Khan
• Created: 28/Aug/20 21:50
• Updated: 28/Aug/20 22:03

Description

Splitting tests originally posted in [PR|https://github.com/apache/spark/pull/29352] for SPARK-32531. The added tests cover cases for maps and arrays of nested structs for different file formats. Eg, https://github.com/apache/spark/pull/29353 and https://github.com/apache/spark/pull/29354 add object reuse when reading ORC and Avro files. However, for dynamic data structures like arrays and maps, we do not know just by looking at the schema what the size of the data structure will be so it has to be allocated when reading the data points. The added tests provide coverage so that objects are not accidentally reused when encountering maps and arrays.

AFAIK this is not covered by existing tests.

Issue Links

• is related to: SPARK-32532 Improve ORC read/write performance on nested structs and array of structs
• is related to: SPARK-32533 Improve Avro read/write performance on nested structs and array of structs
• is related to: SPARK-32531 Add benchmarks for nested structs and arrays for different file formats
• links to: [Github] Pull Request #29573 (msamirkhan)
• links to: [Github] Pull Request #29573 (msamirkhan)