Details
-
Improvement
-
Status: To Do
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
All data sources go through the Enrichment topology, which means that sources should be indexed as they come into the Kafka queue. However, sometimes (for example, during a DDOS attack) one sensor could double or triple in volume for a period of time, swamping the Enrichment topology during that time and preventing data from other sensors from being indexed.
We would like to be able to prioritize data from particular sensors so that even if one sensor is extra volume-heavy, the high-priority sources are still indexed. So if we are running 3 parser topologies, we would like to give them a prioritization so that one gets indexed before the others in the case that the Enrichment topology cannot keep up.