Solr
  1. Solr
  2. SOLR-5886

Store the response in zk for async calls so that it can be returned by REQUESTSTATUS API

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.3, 6.0
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      As of now, only the state of a pre-submitted tasks is returned in the response to the REQUESTSTATUS Collections API call.
      Pass more information, specially in case of a call erroring out.

      1. SOLR-5886.patch
        12 kB
        Anshum Gupta
      2. SOLR-5886.patch
        12 kB
        Anshum Gupta
      3. SOLR-5886-git.patch
        25 kB
        Anshum Gupta
      4. SOLR-5886-git.patch
        18 kB
        Anshum Gupta
      5. SOLR-5986-git.patch
        5 kB
        Anshum Gupta

        Activity

        Hide
        Noble Paul added a comment -

        not just errors. All the output of the non async command should be available for async as well

        Show
        Noble Paul added a comment - not just errors. All the output of the non async command should be available for async as well
        Hide
        Anshum Gupta added a comment -

        sure, makes sense.

        Show
        Anshum Gupta added a comment - sure, makes sense.
        Hide
        Anshum Gupta added a comment -

        Patch that saves the information in zk.
        I'm trying to figure a sensible strategy for limiting the response size (that get's stored in zk) and also trying to make it configurable.

        I am thinking about dropping off the response storage if the number of coreAdmin responses for the call is more than 'x' (configurable).
        If the number of responses is > 'x', No response would be stored and the user would have to look up the log to get information on the task.

        Show
        Anshum Gupta added a comment - Patch that saves the information in zk. I'm trying to figure a sensible strategy for limiting the response size (that get's stored in zk) and also trying to make it configurable. I am thinking about dropping off the response storage if the number of coreAdmin responses for the call is more than 'x' (configurable). If the number of responses is > 'x', No response would be stored and the user would have to look up the log to get information on the task.
        Hide
        Anshum Gupta added a comment -

        Patch updated to trunk and using the new SolrJ collection API calls in the test.

        Show
        Anshum Gupta added a comment - Patch updated to trunk and using the new SolrJ collection API calls in the test.
        Hide
        Anshum Gupta added a comment -

        Updated patch but without the tests. Just running the tests, will upload the patch with tests in a bit.

        Show
        Anshum Gupta added a comment - Updated patch but without the tests. Just running the tests, will upload the patch with tests in a bit.
        Hide
        Noble Paul added a comment -

        How many responses are being stored in ZK?

        Show
        Noble Paul added a comment - How many responses are being stored in ZK?
        Hide
        Anshum Gupta added a comment -

        Right now, there's no limit to that but users need to explicitly clean out the stored responses.

        Having a configurable limit would be a good idea and we could open a separate issue for that.

        Show
        Anshum Gupta added a comment - Right now, there's no limit to that but users need to explicitly clean out the stored responses. Having a configurable limit would be a good idea and we could open a separate issue for that.
        Hide
        Anshum Gupta added a comment -

        Updated patch with tests.

        Show
        Anshum Gupta added a comment - Updated patch with tests.
        Hide
        Noble Paul added a comment -

        Pleas put in an implicit limit of (say 100 items) into the code in this. You can open a ticket to make it configurable later. This is a potentially dangerous feature . Unbound collections are disasters waiting to happen

        Show
        Noble Paul added a comment - Pleas put in an implicit limit of (say 100 items) into the code in this. You can open a ticket to make it configurable later. This is a potentially dangerous feature . Unbound collections are disasters waiting to happen
        Hide
        Anshum Gupta added a comment -

        Again, I don't understand how the limit relates to this change in particular. This only adds more data (actual response) to the same znode.
        #of elements in the map remains the same as before. However, I'll add a limit to restrict storing the response to ~ 500k i.e. only store the response if it's not extremely huge (arbit definition of huge = 500k ? ).

        I'll be happy to have limiting the # of stored statuses as a separate issue.

        Show
        Anshum Gupta added a comment - Again, I don't understand how the limit relates to this change in particular. This only adds more data (actual response) to the same znode. #of elements in the map remains the same as before. However, I'll add a limit to restrict storing the response to ~ 500k i.e. only store the response if it's not extremely huge (arbit definition of huge = 500k ? ). I'll be happy to have limiting the # of stored statuses as a separate issue.
        Hide
        Anshum Gupta added a comment -

        Added limiting the # of stored responses to 10,000.
        The SizeLimitingDistributedMap cleans up 10% of that when that limit is hit.

        Show
        Anshum Gupta added a comment - Added limiting the # of stored responses to 10,000. The SizeLimitingDistributedMap cleans up 10% of that when that limit is hit.
        Hide
        ASF subversion and git services added a comment -

        Commit 1686089 from Anshum Gupta in branch 'dev/trunk'
        [ https://svn.apache.org/r1686089 ]

        SOLR-5886: Store async call results in zk so that they can be returned by the REQUESTSTATUS API. Also, restrict the number of stored response to 10,000 each as a safety net.

        Show
        ASF subversion and git services added a comment - Commit 1686089 from Anshum Gupta in branch 'dev/trunk' [ https://svn.apache.org/r1686089 ] SOLR-5886 : Store async call results in zk so that they can be returned by the REQUESTSTATUS API. Also, restrict the number of stored response to 10,000 each as a safety net.
        Hide
        ASF subversion and git services added a comment -

        Commit 1686101 from Anshum Gupta in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1686101 ]

        SOLR-5886: Store async call results in zk so that they can be returned by the REQUESTSTATUS API. Also, restrict the number of stored response to 10,000 each as a safety net.(merge from trunk)

        Show
        ASF subversion and git services added a comment - Commit 1686101 from Anshum Gupta in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1686101 ] SOLR-5886 : Store async call results in zk so that they can be returned by the REQUESTSTATUS API. Also, restrict the number of stored response to 10,000 each as a safety net.(merge from trunk)
        Hide
        Shalin Shekhar Mangar added a comment -

        Bulk move to 5.4 after 5.3 release.

        Show
        Shalin Shekhar Mangar added a comment - Bulk move to 5.4 after 5.3 release.

          People

          • Assignee:
            Anshum Gupta
            Reporter:
            Anshum Gupta
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development