[HUDI-7086] Scale GCS event source to consume large no of msgs from queue

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.1
• Component/s: deltastreamer
• Labels: pull-request-available
• Assignee: sivabalan narayanan
• Reporter: sivabalan narayanan
• Created: 13/Nov/23 04:48
• Updated: 30/Nov/23 15:30
• Resolved: 30/Nov/23 15:30

Description

We should be able to consume 100 or more messages in one batch for GCS event source. 

 

If apis has some limitations, lets invoke multiple times before ingesting to hudi. 

Issue Links

• links to: GitHub Pull Request #10073
• links to: GitHub Pull Request #10168
• links to: GitHub Pull Request #10171