[NIFI-10660] AvroReader issue after upgrading to Nifi 1.18.0

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.18.0
• Fix Version/s: 1.19.0
• Assignee: Mark Payne
• Reporter: Sandro Berger
• Created: 18/Oct/22 12:08
• Updated: 21/Nov/22 20:09
• Resolved: 21/Nov/22 20:09

Description

After upgrading from 1.15.3 to 1.18.0 we saw some issue regarding AvroReader when the following conditions are met:

1. The AVRO schema in the flowFile uses the "aliases" attribute
2. There is a record in the flowFile where a field with set alias has a value of "null"

The use of aliases and null values used to work for us before upgrading Nifi without issues

Example schema:

{
  "type": "record",
  "name": "test",
  "fields": [
      { "name" : "id", "type" : "int", "aliases" : ["F1"] },
      { "name" : "field1", "type" : [ "string", "null" ], "aliases" : ["F2"] },
      { "name" : "field2", "type" : [ "string", "null" ], "aliases" : ["F3"] }
 ]
}
 

Example data:

[ {
  "id" : 1,
  "field1" : "field1content",
  "field2" : "field2content"
}, {
  "id" : 2,
  "field1" : null,
  "field2" : "field2content"
} ]

Note: If we remove the "aliases" from the schema or make sure that there is never a field with a value of null, we don't have any errors in Nifi 1.18.0.

 

ERROR log:

2022-10-14 16:30:50,617 ERROR [Timer-Driven Process Thread-6] o.a.n.processors.standard.ConvertRecord ConvertRecord[id=c7ee1f0f-10ac-116e-9e4c-300530853ab8] Failed to process StandardFlow
FileRecord[uuid=1d201fdb-e507-4d51-87c2-fbc895a450f3,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1665753427357-4670, container=default, section=574], offset=18384, 
length=324],offset=0,name=1d201fdb-e507-4d51-87c2-fbc895a450f3,size=324]; will route to failure
org.apache.nifi.processor.exception.ProcessException: Could not parse incoming data
        at org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:171)
        at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3434)
        at org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:122)
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1354)
        at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
        at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:59)
        at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.nifi.serialization.MalformedRecordException: Error while getting next record
        at org.apache.nifi.avro.AvroRecordReader.nextRecord(AvroRecordReader.java:50)
        at org.apache.nifi.serialization.RecordReader.nextRecord(RecordReader.java:50)
        at jdk.internal.reflect.GeneratedMethodAccessor150.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
        at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38)
        at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240)
        at com.sun.proxy.$Proxy277.nextRecord(Unknown Source)
        at org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:157)
        ... 13 common frames omitted
Caused by: org.apache.avro.AvroRuntimeException: Not a valid schema field: F2
        at org.apache.avro.generic.GenericData$Record.get(GenericData.java:268)
        at org.apache.nifi.avro.AvroTypeUtil.convertAvroRecordToMap(AvroTypeUtil.java:916)
        at org.apache.nifi.avro.AvroTypeUtil.convertAvroRecordToMap(AvroTypeUtil.java:905)
        at org.apache.nifi.avro.AvroRecordReader.nextRecord(AvroRecordReader.java:43)
        ... 22 common frames omitted 

The error mentions the alias of the field containing the null value.

Could be related to the upgrade of avro introduced with Nifi 1.18.0: https://issues.apache.org/jira/browse/NIFI-10311

Issue Links

• is caused by: AVRO-2278 GenericData.Record field getter not correct
• links to: GitHub Pull Request #6688