[STORM-561] Add ability to create topologies dynamically

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: Flux, storm-core
• Assignee: P. Taylor Goetz
• Reporter: Nathan Leung
• Created: 19/Nov/14 13:08
• Updated: 19/Feb/16 14:53
• Resolved: 03/Jun/15 17:36

Description

It would be nice if a storm topology could be built dynamically, instead of requiring a recompile to change parameters (e.g. number of workers, number of tasks, layout, etc).

I would propose the following data structures for building core storm topologies. I haven't done a design for trident yet but the intention would be to add trident support when core storm support is complete (or in parallel if there are other people working on it):

// fields value and arguments are mutually exclusive
class Argument {
    String argumentType;  // Class used to lookup arguments in method/constructor
    String implementationType; // Class used to create this argument
    String value; // String used to construct this argument
    List<Argument> arguments; // arguments used to build this argument
}

class Dependency {
    String upstreamComponent; // name of upstream component
    String grouping;
    List<Argument> arguments; // arguments for the grouping
}

class StormSpout {
    String name;
    String klazz;  // Class of this spout
    List <Argument> arguments;
    int numTasks;
    int numExecutors;
}

class StormBolt {
    String name;
    String klazz; // Class of this bolt
    List <Argument> arguments;
    int numTasks;
    int numExecutors;
    List<Dependency> dependencies;
}

class StormTopologyRepresentation {
    String name;
    List<StormSpout> spouts;
    List<StormBolt> bolts;
    Map config;
    int numWorkers;
}

Topology creation will be built on top of the data structures above. The benefits:

• Dependency free. Code to unmarshal from json, xml, etc, can be kept in extensions, or as examples, and users can write a different unmarshaller if they want to use a different text representation.
• support arbitrary spout and bolts types
• support of all groupings, streams, via reflections
• ability to specify configuration map via config file
• reification of spout / bolt / dependency arguments
  • recursive argument reification for complex objects

Issue Links

• relates to: STORM-107 Add better ways to construct topologies