Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-3504

TransformCSVToAvro and TransformAvroToCSV incorrectly handle logicalTypes

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Do
    • 1.2.0
    • None
    • Core Framework
    • None

    Description

      The current implementation of the encoding and decoding of Avro logicalTypes is inconsistent with the Avro 1.7.7 and 1.8.1 specifications. Currently it is assumed that a logical type always has the underlying Avro type of Byte. However, that is not consistent with the following text, emphasis mine that was extracted from the logical types section of the Avro specification.

      A logical type is always serialized using its underlying Avro type so that values are encoded in exactly the same way as the equivalent Avro type that does not have a logicalType attribute. Language implementations may choose to represent logical types with an appropriate native type, although this is not required.

      Language implementations must ignore unknown logical types when reading, and should use the underlying Avro type. If a logical type is invalid, for example a decimal with scale greater than its precision, then implementations should ignore the logical type and use the underlying Avro type.

      [1]

      Attachments

        Issue Links

          Activity

            People

              apsaltis Andrew Psaltis
              apsaltis Andrew Psaltis
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: