[SOLR-14673] Add CLI for Streaming Expressions

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: main (10.0)
• Component/s: streaming expressions
• Labels: pull-request-available
• Assignee: Eric Pugh
• Reporter: Joel Bernstein
• Created: 21/Jul/20 12:59
• Updated: Yesterday 12:01
• Resolved: 13/Nov/24 20:33

Description

This ticket will provide a simple CLI that will run a Streaming Expression from the command line and return the results as a delimited result set. This will allow Streaming Expressions to be used from the command line to extract data as well as load data into Solr. 

Sample syntax:

bin/expr expr_file

This will run the expression in expr_file.

Output will be to standard out as delimited records.  

Standard-In, Standard-out and Pipe Composition.

The CLI can read from standard-in and write to standard-out in delimited records. This allows Streaming Expressions to be composed using unix pipes with other command line tools and other streaming expressions. Example syntax:

cat data.csv | ./bin/expr load.expr 

The load.expr file uses the stndin() function to read from standard-in and load date to Solr. Here is a sample load.expr

let(cli-zkhost="localhost:9983",
    
    update(collection1,
           parseCSV(stndin()))
 

In the example above the let expression is used to set the cli-zkhost and then run the Streaming Expression:

update(collection1,
       parseCSV(stndin())

The stndin function reads the data from standard-in. The parseCSV function parses the lines into tuples and the update function sends the data to collection1 for indexing.

 

Querying Solr and Pipe Composition

The CLI can also be used to query Solr and pipe the result to other applications on the command line. This can automate activities like alerting, data backup, replication etc...

 

 

 

 

Issue Links

• incorporates: SOLR-17567 Improve Stream CLI implementation
• is related to: SOLR-16757 Umbrella Ticket for Revamping Solr CLI's for the Future
• links to: GitHub Pull Request #2479