It would be nice if a storm topology could be built dynamically, instead of requiring a recompile to change parameters (e.g. number of workers, number of tasks, layout, etc).
I would propose the following data structures for building core storm topologies. I haven't done a design for trident yet but the intention would be to add trident support when core storm support is complete (or in parallel if there are other people working on it):
Topology creation will be built on top of the data structures above. The benefits:
- Dependency free. Code to unmarshal from json, xml, etc, can be kept in extensions, or as examples, and users can write a different unmarshaller if they want to use a different text representation.
- support arbitrary spout and bolts types
- support of all groupings, streams, via reflections
- ability to specify configuration map via config file
- reification of spout / bolt / dependency arguments
- recursive argument reification for complex objects