Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33316

Support nullable Avro schemas for non-nullable data in Avro writing

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0, 3.0.0, 3.0.1
    • 3.1.0
    • SQL
    • None

    Description

      Currently when users try to use nullable Avro schemas for non-nullable data in Avro writing, Spark will throw a IncompatibleSchemaException.

      There are some cases when users do not have full control over the nullability of the data, or the nullability of the Avro schemas they have to use. We should support nullable Avro schemas for non-nullable data in Avro writing for better usability.

      Attachments

        Activity

          People

            bozhang Bo Zhang
            bozhang Bo Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: