Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
None
-
None
Description
The new ConsumeKafka clients handle the threading model of the consumer api correctly. However, they are not yet honoring partition reassignment cases sufficiently which means we could have avoidable cases of duplication. By registering a partition reassignment listener we can handle it correctly.
Further, the processor is loading subsequent polls of messages into memory rather than writing directly to the process session/disk. We could write them to disk and achieve far better performance and efficiency. Early testing shows easily achieving 100MB/s sustained per thread on a simple laptop setup with defaults which will scale very nicely on a legit installed content repository.
Attachments
Issue Links
- links to
