[NIFI-10448] PublishGCPubSub: not writing fast enough

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.3
• Fix Version/s: 2.0.0-M1, 1.23.0
• Component/s: Core Framework
• Labels: GCP performance
• Environment: VM GCP Centos 7.9 and VM Rocky 8
• Flags: Important
• Assignee: Unassigned
• Reporter: BENABADJI
• Created: 06/Sep/22 16:09
• Updated: 15/Nov/24 15:46
• Resolved: 15/Nov/24 15:46

Description

Hello,

For a need to report IOT message, I cannot write with a PublishGCPubSub processor more than 800 msg/sec despite multiple tuning tested. I am very far from my need with about 60000 msg/sec.
I have an Onpremise NIFI with access through an http proxy, and another nifi on a GCP VM that directly accesses the Topic Pub/Sub.
The versions tested are in 1.16.3 and you will find attached the screenshots with the configuration.
FYI the messages are of the json type and they are less than 1Kb.

Issue Links

• is related to: NIFI-11553 Add Batch Configuration and Record handling for GCP PubSub processors