Uploaded image for project: 'Avro'
  1. Avro
  2. AVRO-2164

Make Decimal a first class type.

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.8.2
    • Fix Version/s: None
    • Component/s: logical types
    • Labels:
      None

      Description

      I'd be interested to hear the communities thoughts on making decimal a first class type. 

      The current logical type encodes a decimal into a bytes or fixed. This encoding does not include any information about the scale, i.e. this encoding is lossy. 

      There are open issues around the compatibility / evolvability of schemas containing decimal logical types, (e.g. AVRO-2078 & AVRO-1721), that mean reading data that was previously written with a different scale will result in data corruption.

      If these issues were fixed, with suitable compatibility checks put in place, this would then make it impossible to evolve an Avro schema where the scale needs to be changed. This inability to evolve the scale is very restrictive, and can result in high overhead for organizations that need to change the scale, i.e. they may potentially need to copy their entire data set, deserializing with the old scale and re-serializing with the new.

      If decimal were promoted to a first class type, this would allow the scale to be captured in the serialized form, allow for schema evolution support.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                BigAndy Andy Coates
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: