Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33316

Support nullable Avro schemas for non-nullable data in Avro writing

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.4.0, 3.0.0, 3.0.1
    • Fix Version/s: 3.1.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Currently when users try to use nullable Avro schemas for non-nullable data in Avro writing, Spark will throw a IncompatibleSchemaException.

      There are some cases when users do not have full control over the nullability of the data, or the nullability of the Avro schemas they have to use. We should support nullable Avro schemas for non-nullable data in Avro writing for better usability.

        Attachments

          Activity

            People

            • Assignee:
              bozhang Bo Zhang
              Reporter:
              bozhang Bo Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: