Details
-
New Feature
-
Status: In Progress
-
Minor
-
Resolution: Unresolved
-
v0.6.0
-
None
-
None
Description
New alert engine architecture:
1)Metaservice: a) get meta(policies,topologies,streamdefinitions and so on)
2)Coordinator: a) pull policies and figure out if polices are changed by Metaservice and notify config consumer of config changes by ConfigBusService (powered by zk node cache)
b) placePolicy depend on policy usage on alertbolt
3)UnitTopology: a) connect CorrelationSpout(read all topics data from kafka) StreamRouterBolt(find which alertbolt to execute) AlertBolt(execute policy on stream) AlertPublisherBolt (publish execute result to db,email and so on)
Let the alert engine to work on spark streaming,I think :
1) #1 and #2 can be worked on both storm and spark platform, can use broadcast to distribute this service
2) we should rewirte CorrelationSpout,StreamRouterBolt,AlertBolt,AlertPublisherBolt to Dstream and connect them as DAG,then submit into spark platform.
3) spark can dynamic allocation executor without restart
4) in the future spark streaming can support add or delete kafka topic without restart
Attachments
Issue Links
1.
|
Supporting Configuration update |
|
Open | JiJun Tang |
2.
|
Kafka Integration without requiring restart of the alert engine |
|
In Progress | JiJun Tang |
3.
|
Supporting Hybird Time Window |
|
In Progress | JiJun Tang |