[NIFI-11863] Postgres COPY command

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Labels: feature improvement
• Assignee: Matt Burgess
• Reporter: John Kuchmek
• Created: 27/Jul/23 15:50
• Updated: 29/Jul/23 18:57

Description

When writing data into Postgres using PutDatabaseRecord the time it takes to commit a large amount of messages (over 1 million) can take a substantial amount of time.  Even with tuning and finding a sweet spot of about 10k records it takes about a minute for the write and commit to happen.  When doing testing, it was found that the Postgres COPY command is about 10 times faster than an INSERT command.  There could be cases where the commit is not needed to be known and speed is all that matters.  

It would be nice to include the ability to use COPY and/or have configurable transactions to help improve performance.