Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9417

Allow daemons to terminate when they finish iterating a topic

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: 6.3
    • Component/s: None
    • Security Level: Public (Default Security Level. Issues are Public)
    • Labels:
      None

      Description

      The daemon expression currently runs until it's killed. This ticket will add a new terminate parameter to the daemon expression that will allow the daemon to shut itself down when it's finished processing a topic.

      There are a couple of small changes that need to be made to allow the daemon to terminate on it's own:

      1) The daemon will need to be passed the Map of all daemons in the /stream handler. This will allow the DaemonStream to remove itself from the Map when it terminates.
      2) Logic needs to be added for the daemon to exit it's run loop if the topic signals it had a zero Tuple run. The sleepMillis value in the EOF Tuple can be used for this purpose. If sleepMillis is greater then 0 then this signals a zero Tuple run.

      1. SOLR-9417.patch
        21 kB
        Joel Bernstein
      2. SOLR-9417.patch
        11 kB
        Joel Bernstein
      3. SOLR-9417.patch
        6 kB
        Joel Bernstein

        Activity

        Hide
        joel.bernstein Joel Bernstein added a comment -

        Initial patch without tests.

        Show
        joel.bernstein Joel Bernstein added a comment - Initial patch without tests.
        Hide
        joel.bernstein Joel Bernstein added a comment -

        Added a very simple test case, which shows a daemon terminating when the underlying topic returns no Tuples. More to follow...

        Show
        joel.bernstein Joel Bernstein added a comment - Added a very simple test case, which shows a daemon terminating when the underlying topic returns no Tuples. More to follow...
        Hide
        joel.bernstein Joel Bernstein added a comment -

        Patch with more robust parallel test. I'll do some manual testing to watch the terminating daemons in action.

        Show
        joel.bernstein Joel Bernstein added a comment - Patch with more robust parallel test. I'll do some manual testing to watch the terminating daemons in action.
        Hide
        joel.bernstein Joel Bernstein added a comment - - edited

        Manual testing is looking good. The following expression worked well and the daemons terminated when they finished processing the topic. So this is the basic template of a batch job, that kicks off then terminates when it's finished:

        parallel(emails1, 
                 workers=2, 
                 sort="daemonOp asc",
                 daemon(id="daemon1",
                        terminate="true",
                        update(emails, 
                               batchSize=250,
                               topic(checkpoints,
                                     emails1,
                                     q="*:*",
                                     id="topic2",
                                     initialCheckpoint=0,
                                     partitionKeys="id",
                                     fl="id, from, to, body"))))
        

        The code above uses parallel workers to iterate over a topic and index the content to another SolrCloud collection.

        Show
        joel.bernstein Joel Bernstein added a comment - - edited Manual testing is looking good. The following expression worked well and the daemons terminated when they finished processing the topic. So this is the basic template of a batch job , that kicks off then terminates when it's finished: parallel(emails1, workers=2, sort= "daemonOp asc" , daemon(id= "daemon1" , terminate= " true " , update(emails, batchSize=250, topic(checkpoints, emails1, q= "*:*" , id= "topic2" , initialCheckpoint=0, partitionKeys= "id" , fl= "id, from, to, body" )))) The code above uses parallel workers to iterate over a topic and index the content to another SolrCloud collection.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit f43742acc5148ea89a9a625818a7229d56e0558e in lucene-solr's branch refs/heads/master from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f43742a ]

        SOLR-9417: Allow daemons to terminate when they finish iterating a topic

        Show
        jira-bot ASF subversion and git services added a comment - Commit f43742acc5148ea89a9a625818a7229d56e0558e in lucene-solr's branch refs/heads/master from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=f43742a ] SOLR-9417 : Allow daemons to terminate when they finish iterating a topic
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 6828dbc9ffac2869238094c738834e541838d886 in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6828dbc ]

        SOLR-9417: Allow daemons to terminate when they finish iterating a topic

        Show
        jira-bot ASF subversion and git services added a comment - Commit 6828dbc9ffac2869238094c738834e541838d886 in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6828dbc ] SOLR-9417 : Allow daemons to terminate when they finish iterating a topic
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 39db5489b9816f7762aad92163b5eb3005331755 in lucene-solr's branch refs/heads/master from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=39db548 ]

        SOLR-9417: Update CHANGES.txt

        Show
        jira-bot ASF subversion and git services added a comment - Commit 39db5489b9816f7762aad92163b5eb3005331755 in lucene-solr's branch refs/heads/master from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=39db548 ] SOLR-9417 : Update CHANGES.txt
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit e61ce95a8fa9c3cf3d42311d59af6a1dd5553eff in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e61ce95 ]

        SOLR-9417: Update CHANGES.txt

        Show
        jira-bot ASF subversion and git services added a comment - Commit e61ce95a8fa9c3cf3d42311d59af6a1dd5553eff in lucene-solr's branch refs/heads/branch_6x from Joel Bernstein [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e61ce95 ] SOLR-9417 : Update CHANGES.txt
        Hide
        shalinmangar Shalin Shekhar Mangar added a comment -

        Closing after 6.3.0 release.

        Show
        shalinmangar Shalin Shekhar Mangar added a comment - Closing after 6.3.0 release.

          People

          • Assignee:
            joel.bernstein Joel Bernstein
            Reporter:
            joel.bernstein Joel Bernstein
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development