[JAMES-3695] Adopt Pulsar as the messaging technology backing the distributed James server

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: master
• Component/s: mailbox, Queue, task
• Labels: full-time gsoc2023 mentor
• Assignee: Jean Helou
• Reporter: Benoit Tellier
• Created: 11/Jan/22 02:13
• Updated: 28/Aug/24 20:51

Description

https://www.mail-archive.com/[e-mail]/msg71462.html

A good long term objective for the PMC is to drop RabbitMQ in
favor of pulsar (third parties could package their own components using
RabbitMQ if they wishes...)

This means:

• Solve the bugs that were found during the Pulsar MailQueue review
• Pulsar MailQueue need to allow listing blobs in order to be
  deduplication friendly.
• Provide an event bus based on Pulsar
• Provide a task manager based on Pulsar
• Package a distributed server backed by pulsar, deprecate then replace
  the current one.
• (optionally) support mail queue priorities

While contributions would of course be welcomed on this topic, we could
offer it as part of GSOC 2022, and we could co-mentor it with mentors of
the Pulsar community (see [3])

[3] https://lists.apache.org/thread/y9s7f6hmh51ky30l20yx0dlz458gw259

Would such a plan gain traction around here ?

Sub-Tasks

• Implements Apache Pulsar based Mailqueue: Resolved
• Solve instable test for the pulsar mail queue: In Progress
• Apache Pulsar mailqueue could support priorities: Open
• Pulsar MailQueue should support deletion of scheduled messages.: Closed
• Pulsar Event Bus implementation: Open
• Dead letter policy for the Pulsar MailQueue: Closed
• Back Task manager components with Apache Pulsar: Open
• Assemble a distributed James server backed by Pulsar: Open
• Pulsar MailQueue: support deduplicating blob store: Open
• Assemble a scaling SMTP server backed by pulsar (and PG): Closed
• Mailqueue filters should not use durable subscriptions: In Progress