[SPARK-43217] Correctly recurse into maps of maps and arrays of arrays in StructType.findNestedField

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: Spark Core
• Assignee: Johan Lasperas
• Reporter: Johan Lasperas
• Created: 20/Apr/23 12:19
• Updated: 24/Apr/23 15:41
• Resolved: 24/Apr/23 15:41

Description

StructType.findNestedField is unable to reach nested fields below two directly nested maps or arrays. Whenever it reaches a map or an array, it'll throw an `invalidFieldName` exception if the child is not a struct.

The following throws 'Field name `a`.`element`.element`.`i` is invalid: `a`.`element`.`element` is not a struct.', even though the access path is valid:

val schema = new StructType()
  .add("a", ArrayType(ArrayType(
    new StructType().add("i", "int"))))
findNestedField(Seq("a", "element", "element", "i"), schema)