[CASSANDRA-18715] Add support for vector search in SAI

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 5.0-alpha2, 5.0, 5.1
• Component/s: Feature/Vector Search
• Epic Link: CEP-30 ANN Vector Search with SAI
• Change Category: Operability
• Complexity: Challenging
• Platform: All

Source Control Link

https://github.com/apache/cassandra/commit/949b760f5516c139591473038917247b1fd7f500

Test and Documentation Plan

Current test run: https://app.circleci.com/pipelines/github/mike-tr-adamson/cassandra/288/workflows/e102cc8b-303a-400e-8265-f49cadf08eb5/jobs/19867

• Assignee: Mike Adamson
• Reporter: Mike Adamson

Authors

Mike Adamson

Reviewers

Andres de la Peña, Jonathan Ellis

• Created: 01/Aug/23 15:58
• Updated: 07/Dec/23 23:18
• Resolved: 29/Oct/23 11:26

Description

The patch associated with this ticket adds a new vector index to SAI. This introduces the following new elements and changes to SAI:

• VectorMemtableIndex - the in-memory representation of the vector indexes that writes data to a DiskANN instance
• VectorSegmentBuilder - that writes a DiskANN graph to the following on-disk components:
  • VECTOR - contains the floating point vectors associated with the graph
  • TERMS - contains the HNSW graph on-disk representation written by a HnswGraphWriter
  • POSTINGS - contains the index postings as written by a VectorPostingsWriter
• VectorIndexSegmentSearcher - used to search the on-disk DiskANN graph

Issue Links

• causes: CASSANDRA-19095 Test Failure: org.apache.cassandra.index.sai.cql.VectorSegmentationTest.testMultipleSegmentsForCompaction
• fixes: CASSANDRA-18927 Fix potential race condition in IndexViewManager during invalidation
• links to: GitHub Pull Request #2673
• links to: GitHub Pull Request #2850