[NIFI-9192] Value of QueryDatabaseTableRecord's "Use Avro Logical Types" property ignored when determining the schema of the recordset

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.15.0
• Component/s: Extensions
• Assignee: Peter Gyori
• Reporter: Peter Gyori
• Created: 03/Sep/21 14:27
• Updated: 28/Sep/21 20:15
• Resolved: 27/Sep/21 20:36

Description

In RecordSqlWriter, the avro schema is determined using this:

final Schema avroSchema = JdbcCommon.createSchema(resultSet, options);

Here, in 'options' the value of "Use Avro Logical Types" is passed and taken into consideration. This avroSchema is used to create the recordAvroSchema, which is OK:

final RecordSchema recordAvroSchema = AvroTypeUtil.createSchema(avroSchema);

However, when creating the fullRecordSet object (which contains the schema as well, the creation:

fullRecordSet = new ResultSetRecordSetWithCallback(resultSet, recordAvroSchema, callback, options.getDefaultPrecision(), options.getDefaultScale())

does not consider the value of the useLogicalTypes flag since it is not passed to the constructor. recordAvroSchema was created considering the value of the flag, and is passed to the constructor but the settings in this schema are overridden during the ResultSetRecordSet creation, where a schema is created based on the metadata received from the database.

As a result, if we have "Use Avro Logical Types" set to false, and we have a column in an Oracle DB table (for the sake of this example) that is of type NUMBER, the resulting avro shcema is:

    "type" : {
      "type" : "bytes",
      "logicalType" : "decimal",
      "precision" : 10,
      "scale" : 0
    }

which unfortunately contains logical avro type.

Issue Links

• links to: GitHub Pull Request #5388