[SPARK-48939] Support recursive reference of Avro schema

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Connect
• Labels: pull-request-available
• Assignee: Yuchen Liu
• Reporter: Yuchen Liu
• Created: 18/Jul/24 23:55
• Updated: 24/Sep/24 23:18
• Resolved: 18/Sep/24 22:45

Description

Recursive reference denotes the case that the type of a field can be defined before in the parent nodes. A simple example is:

{
  "type": "record",
  "name": "LongList",
  "fields" : [
    {"name": "value", "type": "long"},
    {"name": "next", "type": ["null", "LongList"]}
  ]
}

This is written in Avro Schema DSL and represents a linked list data structure. Spark currently will throw an error on this schema. Many users used schema like this, so we should support it. 

Issue Links

• links to: GitHub Pull Request #47425
• links to: GitHub Pull Request #48043