Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-6000

ConvertAvroToORC fails to process Avro type null and rolls back

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.9.0
    • Component/s: Extensions
    • Labels:

      Description

      The ConvertAvroToORC processor throws an IllegalArgumentException when the input avro data has null types or empty arrays.

      ConvertAvroToORC[id=9d22f79d-4ead-3924-df40-2bac4a672055] ConvertAvroToORC[id=9d22f79d-4ead-3924-df40-2bac4a672055] failed to process session due to java.lang.IllegalArgumentException: Did not recognize Avro type null; Processor Administratively Yielded for 1 sec: java.lang.IllegalArgumentException: Did not recognize Avro type null
      java.lang.IllegalArgumentException: Did not recognize Avro type null
      at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.getOrcField(NiFiOrcUtils.java:295)
      at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.lambda$getOrcField$11(NiFiOrcUtils.java:284)
      at java.util.ArrayList.forEach(Unknown Source)
      at org.apache.hadoop.hive.ql.io.orc.NiFiOrcUtils.getOrcField(NiFiOrcUtils.java:281)
      at org.apache.nifi.processors.hive.ConvertAvroToORC.lambda$onTrigger$0(ConvertAvroToORC.java:217)
      at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2910)
      at org.apache.nifi.processors.hive.ConvertAvroToORC.onTrigger(ConvertAvroToORC.java:209)
      at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
      at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162)
      at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:205)
      at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
      at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
      at java.util.concurrent.FutureTask.runAndReset(Unknown Source)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
      at java.lang.Thread.run(Unknown Source)

      The flowfile is not transfered to failure as the https://github.com/apache/nifi/blob/412c4908e2c5d79d958b09403c816db57c828179/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java#L286 only catches ProcessException but https://github.com/apache/nifi/blob/412c4908e2c5d79d958b09403c816db57c828179/nifi-nar-bundles/nifi-hive-bundle/nifi-hive-processors/src/main/java/org/apache/nifi/processors/hive/ConvertAvroToORC.java#L217 Throws IllegalArgumentException when the fieldSchema is anything other than hive primitive types.

        Attachments

        1. AvroToORC_test.xml
          46 kB
          Sujesh Menon

          Issue Links

            Activity

              People

              • Assignee:
                Salatich Aleksandr Salatich
                Reporter:
                sujeshm3 Sujesh Menon
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m