after upgrading our HDF cluster from 220.127.116.11 to 18.104.22.168 our custom processors didn't work anymore and were replaced by "ghost" processors. We had to roll back and it was kind of hard to find out what went wrong.
I think I figured it out now and I'm confused that there are no measures taken or at least documented to avoid this issue. Maybe I didn't find them though, so if that's the case please point them out to me, thanks.
So the root of the problem seems to be this change in nifi: https://github.com/apache/nifi/commit/a27ccd8a56fb52a61d79f87711324142a5b4a141
The corresponding Jira ticket is: https://jira.apache.org/jira/browse/NIFI-5479
What I think happened:
We have our custom processors in a custom library folder configured with "nifi.nar.library.directory.custom=/custom/folder". The last release of our processors happened during 1.7.0 so the NarUnpacker unpacked them into nifis work directory.
Before upgrading to nifi 1.8.0 we went through the changelog and checked whether there were any api incompatibilities, but it didn't seem like it so we thought we would be fine with our current nar.
During the upgrade we saw this log line:
2018-11-30 11:49:52,920 WARN [main] org.apache.nifi.nar.NarClassLoader /var/lib/nifi/work/nar/extensions/custom-nar-1.14.2.nar-unpacked does not contain NAR-INF/bundled-dependencies!
So I guess, the new NarUnpacker didn't re-unpack the .nar file (and thus there was no renaming from META-INF to NAR-INF. Thus the NarClassLoader cannot resolve everything properly.
So a new .nar release/deleting the work directory at this point should allow us to upgrade. But maybe it would be a good idea to make sure this is documented somewhere to save others from some of the frustration and investigation. Maybe it can even be handled (at least for HDF upgrades, but I know that this Jira is the wrong place for that) automatically.
Let me know if you have any questions.