[AVRO-2871] Support more datatypes via logical types plus convenience features

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.9.2
• Component/s: java
• Assignee: Unassigned
• Reporter: Werner Daehn
• Created: 28/Jun/20 07:20
• Updated: 28/Jun/20 07:20

Description

I have created a list of logical datatypes for easier integration with other systems, primarily databases, and wonder if and how others might be interested in that.

Looking for input which of the options you would prefer:

1. I keep it embedded in my project, others cannot use it. No interest by you.
2. Add a library on top of Avro
3. Add it to the Avro library, which helps registering the logical types automatically

Any advice?

https://github.com/rtdi/connectorbase/tree/master/pipelinefoundation/src/main/java/io/rtdi/bigdata/connector/pipeline/foundation/avrodatatypes

Note: Code needs some cleanup.

 

Detailed feature

Currently there is quite a break between native datatypes and logical datatypes. For example the user has to read the schema, then find the Avro datatype, then check if a logical datatype is defined on top of it.

Also setting values is coding wise apparently very simple with a record.put("REVENUE", 1234.56) but fails at runtime as the decimal is not a BigDecimal converted into a byte array.

If all datatypes are logical datatypes, this can be made simpler for the user.

Hence I created logical datatypes for the base datatypes plus all common database datatypes. Examples are NVARCHAR(100), CLOB, ST_POINT, VARCHAR(100),... all the usual candidates needed for databases.

 

The purpose of these logical types are:

• Richer metadata
• Simpler data assignment (e,g, decimal datatype can get a Java int, biginteger, bigdecimal,..)
• Simpler datatype mapping from Avro-to-JDBC and vice versa
• One code line required for finding the exact datatype