[SOLR-9384] Add randomization to the train Streaming Expression to support very large training sets

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: streaming expressions
• Assignee: Unassigned
• Reporter: Joel Bernstein
• Created: 04/Aug/16 16:58
• Updated: 14/Aug/21 20:06

Description

The train (SOLR-9252) Streaming Expression optimizes a logistic regression model on text.

The initial implementation instantiates a doc vector for each document in the training set on each iteration. The doc vectors are held in memory so, the size of the training set is limited by memory constraints.

This ticket will add randomization to the algorithm so that a random set of documents from the training set are processed on each iteration.

This will allow the train Streaming Expression to be run on much larger training sets.