Solr
  1. Solr
  2. SOLR-2908

To push the terms.limit parameter from the master core to all the shard cores.

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Critical Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.4.1
    • Fix Version/s: 4.9, 5.0
    • Labels:
    • Environment:

      Linux server. 64 bit processor and 16GB Ram.

      Description

      When we pass the terms.limit parameter to the master (which has many shard cores), it's not getting pushed down to the individual cores. Instead the default value of -1 is assigned to Terms.limit parameter is assigned in the underlying shard cores. The issue being the time taken by the Master core to return the required limit of terms is higher when we are having more number of underlying shard cores. This affects the performances of the auto suggest feature.

      Can thought we can have a parameter to explicitly override the -1 being set to Terms.limit in shards core.

      We saw the source code(TermsComponent.java) and concluded that the same. Please help us in pushing the terms.limit parameter to shard cores.

      PFB code snippet.

      private ShardRequest createShardQuery(SolrParams params)

      { ShardRequest sreq = new ShardRequest(); sreq.purpose = ShardRequest.PURPOSE_GET_TERMS; // base shard request on original parameters sreq.params = new ModifiableSolrParams(params); // remove any limits for shards, we want them to return all possible // responses // we want this so we can calculate the correct counts // dont sort by count to avoid that unnecessary overhead on the shards sreq.params.remove(TermsParams.TERMS_MAXCOUNT); sreq.params.remove(TermsParams.TERMS_MINCOUNT); sreq.params.set(TermsParams.TERMS_LIMIT, -1); sreq.params.set(TermsParams.TERMS_SORT, TermsParams.TERMS_SORT_INDEX); return sreq; }

      Solr Version:
      Solr Specification Version: 1.4.0.2010.01.13.08.09.44
      Solr Implementation Version: 1.5-dev exported - yonik - 2010-01-13 08:09:44
      Lucene Specification Version: 2.9.1-dev
      Lucene Implementation Version: 2.9.1-dev 888785 - 2009-12-09 18:03:31

      1. SOLR-2908.patch
        6 kB
        Vitaliy Zhovtyuk
      2. SOLR-2908.patch
        5 kB
        Vitaliy Zhovtyuk

        Activity

        sivaganesh created issue -
        sivaganesh made changes -
        Field Original Value New Value
        Description When we pass the terms.limit parameter to the master (which has many shard cores), it's not getting pushed down to the individual cores. Instead the default value of -1 is assigned to Terms.limit parameter is assigned in the underlying shard cores. The issue being the time taken by the Master core to return the required limit of terms is higher when we are having more number of underlying shard cores. This affects the performances of the auto suggest feature.

        Can thought we can have a parameter to explicitly override the -1 being set to Terms.limit in shards core.

        We saw the source code(TermsComponent.java) and concluded that the same. Please help us in pushing the terms.limit parameter to shard cores.

        PFB code snippet.

        private ShardRequest createShardQuery(SolrParams params) {
            ShardRequest sreq = new ShardRequest();
            sreq.purpose = ShardRequest.PURPOSE_GET_TERMS;

            // base shard request on original parameters
            sreq.params = new ModifiableSolrParams(params);

            // remove any limits for shards, we want them to return all possible
            // responses
            // we want this so we can calculate the correct counts
            // dont sort by count to avoid that unnecessary overhead on the shards
            sreq.params.remove(TermsParams.TERMS_MAXCOUNT);
            sreq.params.remove(TermsParams.TERMS_MINCOUNT);
            sreq.params.set(TermsParams.TERMS_LIMIT, -1);
            sreq.params.set(TermsParams.TERMS_SORT, TermsParams.TERMS_SORT_INDEX);

            return sreq;
          }
        When we pass the terms.limit parameter to the master (which has many shard cores), it's not getting pushed down to the individual cores. Instead the default value of -1 is assigned to Terms.limit parameter is assigned in the underlying shard cores. The issue being the time taken by the Master core to return the required limit of terms is higher when we are having more number of underlying shard cores. This affects the performances of the auto suggest feature.

        Can thought we can have a parameter to explicitly override the -1 being set to Terms.limit in shards core.

        We saw the source code(TermsComponent.java) and concluded that the same. Please help us in pushing the terms.limit parameter to shard cores.

        PFB code snippet.

        private ShardRequest createShardQuery(SolrParams params) {
            ShardRequest sreq = new ShardRequest();
            sreq.purpose = ShardRequest.PURPOSE_GET_TERMS;

            // base shard request on original parameters
            sreq.params = new ModifiableSolrParams(params);

            // remove any limits for shards, we want them to return all possible
            // responses
            // we want this so we can calculate the correct counts
            // dont sort by count to avoid that unnecessary overhead on the shards
            sreq.params.remove(TermsParams.TERMS_MAXCOUNT);
            sreq.params.remove(TermsParams.TERMS_MINCOUNT);
            sreq.params.set(TermsParams.TERMS_LIMIT, -1);
            sreq.params.set(TermsParams.TERMS_SORT, TermsParams.TERMS_SORT_INDEX);

            return sreq;
          }

        Solr Version:
        Solr Specification Version: 1.4.0.2010.01.13.08.09.44
         Solr Implementation Version: 1.5-dev exported - yonik - 2010-01-13 08:09:44
         Lucene Specification Version: 2.9.1-dev
         Lucene Implementation Version: 2.9.1-dev 888785 - 2009-12-09 18:03:31

        Mark Miller made changes -
        Fix Version/s 4.1 [ 12321141 ]
        Fix Version/s 5.0 [ 12321664 ]
        Fix Version/s 1.4.1 [ 12315096 ]
        Mark Miller made changes -
        Fix Version/s 4.2 [ 12323893 ]
        Fix Version/s 4.1 [ 12321141 ]
        Robert Muir made changes -
        Fix Version/s 4.3 [ 12324128 ]
        Fix Version/s 5.0 [ 12321664 ]
        Fix Version/s 4.2 [ 12323893 ]
        Uwe Schindler made changes -
        Fix Version/s 4.4 [ 12324324 ]
        Fix Version/s 4.3 [ 12324128 ]
        Steve Rowe made changes -
        Fix Version/s 5.0 [ 12321664 ]
        Fix Version/s 4.5 [ 12324743 ]
        Fix Version/s 4.4 [ 12324324 ]
        Adrien Grand made changes -
        Fix Version/s 4.6 [ 12325000 ]
        Fix Version/s 5.0 [ 12321664 ]
        Fix Version/s 4.5 [ 12324743 ]
        Uwe Schindler made changes -
        Fix Version/s 4.7 [ 12325573 ]
        Fix Version/s 4.6 [ 12325000 ]
        Vitaliy Zhovtyuk made changes -
        Attachment SOLR-2908.patch [ 12629092 ]
        Shalin Shekhar Mangar made changes -
        Assignee Shalin Shekhar Mangar [ shalinmangar ]
        Vitaliy Zhovtyuk made changes -
        Attachment SOLR-2908.patch [ 12629789 ]
        David Smiley made changes -
        Fix Version/s 4.8 [ 12326254 ]
        Fix Version/s 4.7 [ 12325573 ]
        Uwe Schindler made changes -
        Fix Version/s 4.9 [ 12326731 ]
        Fix Version/s 5.0 [ 12321664 ]
        Fix Version/s 4.8 [ 12326254 ]

          People

          • Assignee:
            Shalin Shekhar Mangar
            Reporter:
            sivaganesh
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Time Tracking

              Estimated:
              Original Estimate - 168h
              168h
              Remaining:
              Remaining Estimate - 168h
              168h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development