[OAK-9004] Multitenant index name strategy with reindex support

• Type: Task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.30.0
• Component/s: indexing
• Assignee: Unassigned
• Reporter: Amrit Verma
• Created: 10/Apr/20 04:37
• Updated: 27/May/20 06:19
• Resolved: 16/Apr/20 15:52

Description

Index names in elastic follow the pattern <name>-<reindex-count>

This has to be changed in order to support multi-tenancy.

We need to pass a parameter with the customer ID so we can create indexes like <customer>.<name>-<reindex-count>.  ES indexes cannot be longer than 255 bytes and must comply with the following criteria https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params

 

We need also to decide what to do on re-index. The simpler strategy would be to:

• create the new index
• move writes and reads to the new index
• delete old index

A more sophisticated strategy could:

• create the new index
• move writes to the new index
• reads will continue to use the old index until the new one catches up
• when the new one is in sync, move reads to the new index & delete the old one

Both strategies can be implemented using Aliases in Elasticsearch to avoid race conditions. To implement the second solution we need something that tells us when the new index has caught up with the initial load.

Issue Links

• is cloned by: OAK-9053 Reindexing Strategy for ES indexes