Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
1.8.0, 1.7.1
-
None
Description
We are using a LookupRecord processor to add an array property to a Record. The LookupRecord's `Lookup Service` is a MongoDBLookupService. The idea is for the LookupRecord to query Mongo documents by the Record's `_id`, to extract the array property at `f` on that document, and to insert that array into the Record at the RecordPath `f`. This flow results in the conversion error copied below, which completely prevents the FlowFile from proceeding (routes to Failure/Unmatched.) It appears that the `MongoDBLookupService` (or perhaps its `MongoDBControllerService`) stores Mongo document arrays as `ArrayList`s, which `DataTypeUtils#toArray` does not support. A simple fix appears to be to add a few lines to `DataTypeUtils#toArray` to support `ArrayList`s. I will provide a PR for this issue.
This issue appears in 1.7.1, and is extant in 1.8.0-SNAPSHOT (2018-10-02 b4c8e0179). I am not aware of any workaround to enrich a Record with an array from Mongo.
LookupRecord Configuration
Property | Value |
---|---|
Result RecordPath | /f |
Record Result Contents | Insert Entire Record |
_id | /_id |
MongoDBLookupService Configuration
Property | Value |
---|---|
Lookup Value Field | f |
Conversion error (with irrelevant Mongo document and schema properties elided):
2018-10-04 15:00:31,348 ERROR [Timer-Driven Process Thread-1] o.a.n.processors.standard.LookupRecord LookupRecord[id=339bf72c-91c3-36f0-ac14-bc8b734e2582] Failed to write MapRecord[{... f=[Document{{fid=5762addaffa7cf0980267acb, l=1.0}}, Document{{fid=5762adf32fc74d0418104201, l=2.12}}, Document{{fid=5762addd18b8b607802fec6f, l=3.32}}, Document{{fid=587542dd638e870838a79d08, l=1.0}}, Document{{fid=5762addbffa7cf0980267acd, l=3.32}}], i=[Ljava.lang.Object;@35dee8fc, ...] with schema { "namespace": "nifi", "name": "MongoPosting", "type": "record", "fields": [ ... { "name": "f", "type": [ "null", { "type": "array", "items": { "type": "record", "name": "PostingFeatureDetails", "fields": [ { "name": "fid", "type": "string" }, { "name": "l" , "type": ["null", "double"] } ]} } ]}, ... ] } as a JSON Object due to org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}}, Document{{fid=5762adf32fc74d0418104201, l=2.12}}, Document{{fid=5762addd18b8b607802fec6f, l=3.32}}, Document{{fid=587542dd638e870838a79d08, l=1.0}}, Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class java.util.ArrayList to Object Array for field f: org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}}, Document{{fid=5762adf32fc74d0418104201, l=2.12}}, Document{{fid=5762addd18b8b607802fec6f, l=3.32}}, Document{{fid=587542dd638e870838a79d08, l=1.0}}, Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class java.util.ArrayList to Object Array for field f org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}}, Document{{fid=5762adf32fc74d0418104201, l=2.12}}, Document{{fid=5762addd18b8b607802fec6f, l=3.32}}, Document{{fid=587542dd638e870838a79d08, l=1.0}}, Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class java.util.ArrayList to Object Array for field f at org.apache.nifi.serialization.record.util.DataTypeUtils.toArray(DataTypeUtils.java:347) at org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:153) at org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:115) at org.apache.nifi.json.WriteJsonResult.writeValue(WriteJsonResult.java:284) at org.apache.nifi.json.WriteJsonResult.writeRecord(WriteJsonResult.java:187) at org.apache.nifi.json.WriteJsonResult.writeRecord(WriteJsonResult.java:136) at org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59) at org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:148) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186) at org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2018-10-04 15:00:31,349 ERROR [Timer-Driven Process Thread-1] o.a.n.processors.standard.LookupRecord LookupRecord[id=339bf72c-91c3-36f0-ac14-bc8b734e2582] Failed to process StandardFlowFileRecord[uuid=29b498fe-787f-4c62-a237-f95a8ebcb0b2,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1538665215854-2, container=default, section=2], offset=100486, length=635840],offset=624238,name=1675395032391,size=11602]: org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}}, Document{{fid=5762adf32fc74d0418104201, l=2.12}}, Document{{fid=5762addd18b8b607802fec6f, l=3.32}}, Document{{fid=587542dd638e870838a79d08, l=1.0}}, Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class java.util.ArrayList to Object Array for field f org.apache.nifi.serialization.record.util.IllegalTypeConversionException: Cannot convert value [[Document{{fid=5762addaffa7cf0980267acb, l=1.0}}, Document{{fid=5762adf32fc74d0418104201, l=2.12}}, Document{{fid=5762addd18b8b607802fec6f, l=3.32}}, Document{{fid=587542dd638e870838a79d08, l=1.0}}, Document{{fid=5762addbffa7cf0980267acd, l=3.32}}]] of type class java.util.ArrayList to Object Array for field f at org.apache.nifi.serialization.record.util.DataTypeUtils.toArray(DataTypeUtils.java:347) at org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:153) at org.apache.nifi.serialization.record.util.DataTypeUtils.convertType(DataTypeUtils.java:115) at org.apache.nifi.json.WriteJsonResult.writeValue(WriteJsonResult.java:284) at org.apache.nifi.json.WriteJsonResult.writeRecord(WriteJsonResult.java:187) at org.apache.nifi.json.WriteJsonResult.writeRecord(WriteJsonResult.java:136) at org.apache.nifi.serialization.AbstractRecordSetWriter.write(AbstractRecordSetWriter.java:59) at org.apache.nifi.processors.standard.AbstractRouteRecord$1.process(AbstractRouteRecord.java:148) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2218) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2186) at org.apache.nifi.processors.standard.AbstractRouteRecord.onTrigger(AbstractRouteRecord.java:121) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Attachments
Attachments
Issue Links
- is duplicated by
-
NIFI-5731 Support List-typed fields for Array coercion
- Resolved
- links to
GitHub user snagacarl opened a pull request:
https://github.com/apache/nifi/pull/3049
NIFI-5664Support ArrayList in DataTypeUtils#toArrayThank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
in the commit message?
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Snagajob/nifi
NIFI-5664Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/3049.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3049
commit 30d0410f8f35cff772cc74fc36e4368d33310aec
Author: Carl Gieringer <carl.gieringer@...>
Date: 2018-10-04T16:50:08Z
NIFI-5664Support ArrayList in DataTypeUtils#toArray