[SPARK-39802] Support recursive references in Avro schemas in Spark

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.4.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Ivan Sadikov
• Created: 18/Jul/22 00:50
• Updated: 29/Jul/22 17:20

Description

This is a follow-up for https://issues.apache.org/jira/browse/SPARK-25718. 

It would be great if Spark could support recursive references in Avro schemas as currently this is disabled and the following exception is thrown: 

org.apache.spark.sql.avro.IncompatibleSchemaException: Found recursive reference in Avro schema, which can not be processed by Spark: {
  "type": "record",
  "name": "Struct",
  "fields": [
    {
      "name": "fields",
      "type": [
        "null",
        {
          "type": "array",
          "items": {
            "type": "record",
            "name": "fields",
            "fields": [
              {
                "name": "value",
                "type": [
                  "null",
                  "string"
                ],
                "default": null
              }
            ]
          }
        }
      ],
      "default": null
    }
  ]
}

I would appreciate it if someone could comment on the overall effort to make recursive Avro schemas work in Spark and whether or not there are any API restrictions that could potentially hinder the implementation, thanks.

Issue Links

• is related to: SPARK-25718 Detect recursive reference in Avro schema and throw exception