[SOLR-17065] Migrate Sandbox CrossDC functionality into Solr

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: CrossDC, module - crossDC
• Labels: pull-request-available
• Assignee: Houston Putman
• Reporter: Houston Putman
• Created: 03/Nov/23 16:09
• Updated: 01/Oct/24 17:36

Description

As a part of SIP-13: Cross Data Center Replication, the solr-sandbox repository was created and a new Cross DC implementation has been developed.

This CrossDC implementation relies on Kafka (for now, maybe other queues in the future), and has two parts.

• A producer, which is a Solr plugin (An updateRequestProcessor), that sends documents to kafka after successfully indexing them.
• A consumer, which is a standalone application, that reads documents for kafka and sends them to the mirrored SolrClouds.

This has been in development for a year now and is used at multiple companies (even in production).

We believe that the project has reached a level of maturity that it can be "graduated" from the sandbox module and be included in future Solr releases.

 

As a part of the move I propose the following:

• The Producer will be branded as the "cross-dc" Solr module, and be available just as any other module is.
• The Consumer will be branded as the "cross-dc-manager" and be packaged just as the prometheus-exporter is, a standalone application that comes with Solr and uses many of the same Jars. The reason for the name change is that we might expand the role of this application to something beyond just "consuming" from the Kafka queue. In that case its much easier to rename it now than later.

 

We will let this bake in the main branch for a while before backporting to 9.x.

Issue Links

• links to: GitHub Pull Request #2062