Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
3.2.12
-
None
-
None
-
None
Description
In theory, this can be achieved in two different ways:
1) Update code to support concurrent PeriodicNotificationApplications in separate YARN containers that split the workload for a single rya instance.
2) Update code to support the various Runnables executing in separate YARN containers. This would likely mean replacing the blocking queue being used with serialization and kafka, or possibly adopting the kafka streams library.
The second option has the benefit that it allows for finer grained control of concurrency, though it does introduce an added serialization overhead.
Other goal for this effort should be to ensure graceful degradation of performance in the event of a container failure.