Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-7268

Add a script to pipe data from other programs or files to Solr using SolrJ

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: scripts and tools, SolrJ
    • Labels:
      None

      Description

      I should be able to pipe JSON/XML/CSV or whatever is possible at the /update/* to a command which in turn uses SolrJ to send the docs to the correct leader in native format.
      In the following examples , all connection details of the cluster is put into a file called solrj.properties
      example :

      #post a file
      cat myjson.json | bin/post -c gettingstarted -s http://localhost:8983/solr 
      #or a producer program
      myprogram | bin/post  -c gettingstarted -s http://localhost:8983/solr
      

      The behavior of the script would be exactly similar to the behavior if I were to post the request directly to solr to the specified qt . Everything parameter the requesthandler accepts would be accepted as a -<param-name>=<param-value> format. The same things could be put into a properties file called indexer.properties and be passed as a -p parameter. The script would expect the following extra properties zk.url for cloud or solr.url for standalone.

        Activity

        Hide
        janhoy Jan Høydahl added a comment -

        post.jar already reads from stdin if you pass -Ddata=stdin. Does not use SolrJ though, but perhaps it is time for bin/post to start using SolrJ?

        The open-ended -<param-name>=<param-value> is scary if a request handler's param overlaps with script args.

        Show
        janhoy Jan Høydahl added a comment - post.jar already reads from stdin if you pass -Ddata=stdin . Does not use SolrJ though, but perhaps it is time for bin/post to start using SolrJ? The open-ended -<param-name>=<param-value> is scary if a request handler's param overlaps with script args.
        Hide
        noble.paul Noble Paul added a comment -

        you are right the -<param-name>=<param-value> can lead to conflicts we can just have a generic param like -params key1=val1&key2=val2 etc. Anyway , nobody has yet picked up the implementation

        Show
        noble.paul Noble Paul added a comment - you are right the -<param-name>=<param-value> can lead to conflicts we can just have a generic param like -params key1=val1&key2=val2 etc. Anyway , nobody has yet picked up the implementation

          People

          • Assignee:
            noble.paul Noble Paul
            Reporter:
            noble.paul Noble Paul
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development