[SPARK-25718] Detect recursive reference in Avro schema and throw exception

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: SQL
• Assignee: Gengliang Wang
• Reporter: Gengliang Wang
• Created: 12/Oct/18 15:16
• Updated: 13/Oct/18 06:51
• Resolved: 13/Oct/18 06:51

Description

Avro schema allows recursive reference, e.g. the example schema in https://avro.apache.org/docs/1.8.2/spec.html#schema_record
```
{
"type": "record",
"name": "LongList",
"aliases": ["LinkedLongs"], // old name for this
"fields" : [

{"name": "value", "type": "long"}

, // each element has a long

{"name": "next", "type": ["null", "LongList"]}

// optional next element
]
}
```

However, it is impossible to convert the schema as `StructType` in current Spark SQL. Running `SchemaConverters.toSqlType(avroSchema: Schema)` and we will get stack overflow exception.

We should detect the recursive reference and throw exception for it.

Issue Links

• links to: [Github] Pull Request #22709 (gengliangwang)
• links to: [Github] Pull Request #22709 (gengliangwang)