Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
1.16.1, 1.16.2, 1.16.3
-
None
Description
A change was implemented in NIFI-9903 which results in lookupService.lookup() being called twice per record (at least until the first match). For lookup services which are idempotent (CSVRecordLookupService, DistributedMapCacheLookupService, PropertiesFileLookupService) making lookups twice won’t affect the result or have undesired side effects. However, the RestLookupService can make arbitrary HTTP requests for the standard HTTP methods (GET, POST, PUT, DELETE) and there is no guarantee that these requests will be idempotent. POST requests in particular are not expected to be idempotent and may cause undesirable behaviour if invoked multiple times (as in our case).
As the name suggests, LookupRecord could be expected to be used only to perform lookups which are idempotent and do not have side effects. Matt Burgess wrote an article where it seems the expected behaviour was that lookupService.lookup() would only be called once. The change in behaviour and being called twice would now cause IDs to be skipped.
It was suggested by Mark Payne in a Slack discussion that lookup results could be cached up until the first match, which may alleviate the issues we are seeing.
