Description
Avro schema allows recursive reference, e.g. the example schema in https://avro.apache.org/docs/1.8.2/spec.html#schema_record
{
"type": "record",
"name": "LongList",
"aliases": ["LinkedLongs"], // old name for this
"fields" : [
{"name": "value", "type": "long"}, // each element has a long
{"name": "next", "type": ["null", "LongList"]} // optional next element
]
}
However, it is impossible to convert the schema as `StructType` in current Spark SQL. Running `SchemaConverters.toSqlType(avroSchema: Schema)` and we will get stack overflow exception.
We should detect the recursive reference and throw exception for it.
Attachments
Issue Links
- relates to
-
SPARK-39802 Support recursive references in Avro schemas in Spark
-
- Open
-
- links to
