[NIFI-11621] Inferring schema for JSON fails when there's a CHOICE of different ARRAY types

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0-M1, 1.22.0
• Component/s: Extensions
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 31/May/23 18:14
• Updated: 31/May/23 21:08
• Resolved: 31/May/23 21:08

Description

From Apache Slack: https://apachenifi.slack.com/archives/C0L9VCD47/p1685553667778359?thread_ts=1685461745.470939&cid=C0L9VCD47

When using ConvertRecord with a JSON Reader and an Avro Writer, when inferring the JSON schema, each of the following two records works properly:

{"test_record":{"array_test_record":{"test_array":[]}}}

{"test_record":{"array_test_record":{"test_array":["test"]}}}

However, when combined into a single FlowFile:

{"test_record":{"array_test_record":{"test_array":[]}}}
{"test_record":{"array_test_record":{"test_array":["test"]}}}

It fails with a NullPointerException:

2023-05-31 13:51:35,632 ERROR [Timer-Driven Process Thread-8] o.a.n.processors.standard.ConvertRecord ConvertRecord[id=72e564dc-0188-1000-360a-9f86b50ec8ac] Failed to process StandardFlowFileRecord[uuid=9bf4f0fb-0942-48ba-8a16-0dbd98db3f97,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1685554864966-1, container=default, section=1], offset=3278, length=117],offset=0,name=9bf4f0fb-0942-48ba-8a16-0dbd98db3f97,size=117]; will route to failure
org.apache.nifi.processor.exception.ProcessException: Could not determine the Avro Schema to use for writing the content
        at org.apache.nifi.avro.AvroRecordSetWriter.createWriter(AvroRecordSetWriter.java:154)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254)
        at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:105)
        at com.sun.proxy.$Proxy177.createWriter(Unknown Source)
        at org.apache.nifi.processors.standard.AbstractRecordProcessor$1.process(AbstractRecordProcessor.java:150)
        at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:3441)
        at org.apache.nifi.processors.standard.AbstractRecordProcessor.onTrigger(AbstractRecordProcessor.java:122)
        at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1360)
        at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:243)
        at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
        at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.nifi.schema.access.SchemaNotFoundException: Failed to compile Avro Schema
        at org.apache.nifi.avro.AvroRecordSetWriter.createWriter(AvroRecordSetWriter.java:145)
        ... 21 common frames omitted
Caused by: java.lang.NullPointerException: null
        at org.apache.nifi.avro.AvroTypeUtil.buildAvroSchema(AvroTypeUtil.java:208)
        at org.apache.nifi.avro.AvroTypeUtil.buildAvroField(AvroTypeUtil.java:130)
        at org.apache.nifi.avro.AvroTypeUtil.buildAvroSchema(AvroTypeUtil.java:284)
        at org.apache.nifi.avro.AvroTypeUtil.buildAvroSchema(AvroTypeUtil.java:241)
        at org.apache.nifi.avro.AvroTypeUtil.buildAvroField(AvroTypeUtil.java:130)
        at org.apache.nifi.avro.AvroTypeUtil.buildAvroSchema(AvroTypeUtil.java:284)
        at org.apache.nifi.avro.AvroTypeUtil.buildAvroField(AvroTypeUtil.java:130)
        at org.apache.nifi.avro.AvroTypeUtil.buildAvroSchema(AvroTypeUtil.java:122)
        at org.apache.nifi.avro.AvroTypeUtil.extractAvroSchema(AvroTypeUtil.java:102)
        at org.apache.nifi.avro.AvroRecordSetWriter.createWriter(AvroRecordSetWriter.java:142)
        ... 21 common frames omitted

Issue Links

• links to: GitHub Pull Request #7322