[NIFI-9295] Issue with PutMongo(and PutMongoRecord)

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.11.4
• Component/s: Configuration
• Assignee: Unassigned
• Reporter: Deepak Reddy Chirthani
• Created: 12/Oct/21 17:20
• Updated: 12/Oct/21 17:20

Description

We are having an issue with PutMongoRecord Processor(same for PutMongo as well) which is routing the flowfiles to the failure relationship when the Mongo database(or the server hosting the db) is down. We have a set-up which looks like below on a high-level overview.

As you can clearly see, all the failure relationships of different processors in the dataflow are connected to an Error Handling Process Group. Now, when the Mongodb is down, the PutMongoRecord processor is routing all the flowfiles to the failure relationship. Now, I know that a processor could route a flowfile to the failure relationship for multitude of reasons and I believe PutMongoRecord routing the flowfiles to failure when the database is down is not ideal for our scenario. Our dataflow is real-time streaming and when mongo database is down and flowfiles get routed to failure relationship, we get hundreds of emails because of the PutEmail processor in the Error Handling Process Group. We configured the Error Handling process group to check if we get any erroneous data which is against our schema or whatever maybe the reason.

Is there any logic you guys help us in this scenario?

I also observed that when we load the data to a RDBMS, lets say SQL Server, using the PutSql processor, it doesn’t route the flowfiles to the failure relationship when the sql server is down. I remember the flowfiles were held up in the connector above it until the db is up and running. I also noticed that PutSql has retry and PutMongo(and Record) doesn’t have any retry.

Thanks in Advance. Please comment if you guys have any questions