[STORM-2147] [Storm SQL] Support automatic spout parallelism based on DataSource metadata

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Abandoned
• Component/s: storm-sql
• Labels: storm-sql-optimaztion
• Assignee: Unassigned
• Reporter: Jungtaek Lim
• Created: 14/Oct/16 23:15
• Updated: 02/Jul/24 09:55
• Resolved: 02/Jul/24 09:55

Description

It would be better to receive metadata from Data Source, especially producer which can give some hints to optimize.
A notable kind of hint is parallelism hint. In storm-kafka we know that normally it's best to set parallelism to same as topic's partition count so that Spouts can pull the data from all partitions in parallel.

We can apply non-query optimizations start from here.

Issue Links

• mentioned in: Page Loading...