Solr
  1. Solr
  2. SOLR-3527

SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas.

    Details

      Description

      Send the following command to a Solr server with many segments in a multi-shard, multi-server environment:

      curl "http://localhost:8080/solr/update?optimize=true&waitFlush=true&maxSegments=6&distrib=false"

      The local server will end up with the number of segments at 6, as requested, but all other shards in the index will be optimized with maxSegments=1, which takes far longer to complete. All shards should be optimized to the requested value of 6.

        Activity

        Hide
        Andy Laird added a comment -

        One additional data point: the distrib=false does not matter with current behavior. It seems if distrib=false only the local server should be optimized (to the requested value) and if distrib=true (default) all shards in the index should be optimized with N max segments.

        Show
        Andy Laird added a comment - One additional data point: the distrib=false does not matter with current behavior. It seems if distrib=false only the local server should be optimized (to the requested value) and if distrib=true (default) all shards in the index should be optimized with N max segments.
        Hide
        Mark Miller added a comment -

        Sounds right Andy - thanks for the report.

        Show
        Mark Miller added a comment - Sounds right Andy - thanks for the report.
        Hide
        Tomás Fernández Löbbe added a comment - - edited

        I was investigating this issue and found that DistribUpdateProcessor distributes the commit commands using the SolrCommandDistributor.distribCommit(...). The method creates Actions to be distributed to other nodes like:

        void addCommit(UpdateRequestExt ureq, CommitUpdateCommand cmd) {
            if (cmd == null) return;
            ureq.setAction(cmd.optimize ? AbstractUpdateRequest.ACTION.OPTIMIZE
                : AbstractUpdateRequest.ACTION.COMMIT, false, cmd.waitSearcher);
          }
        

        In that method, the action is not considering the "maxOptimizeSegments" parameter, that's why it's not being distributed to other nodes. However, a bigger problem may be that this method is omitting other parameters too, like "softCommit" and "epungeDeletes", which means that an explicit soft commit issued like:

        http://host:port/solr/update?commit=true&softCommit=true
        

        can be distributed as a hard commit to the nodes. I'm not sure about this because I haven't write any test case yet, but it's definitely something to test.

        Show
        Tomás Fernández Löbbe added a comment - - edited I was investigating this issue and found that DistribUpdateProcessor distributes the commit commands using the SolrCommandDistributor.distribCommit(...). The method creates Actions to be distributed to other nodes like: void addCommit(UpdateRequestExt ureq, CommitUpdateCommand cmd) { if (cmd == null ) return ; ureq.setAction(cmd.optimize ? AbstractUpdateRequest.ACTION.OPTIMIZE : AbstractUpdateRequest.ACTION.COMMIT, false , cmd.waitSearcher); } In that method, the action is not considering the "maxOptimizeSegments" parameter, that's why it's not being distributed to other nodes. However, a bigger problem may be that this method is omitting other parameters too, like "softCommit" and "epungeDeletes", which means that an explicit soft commit issued like: http: //host:port/solr/update?commit= true &softCommit= true can be distributed as a hard commit to the nodes. I'm not sure about this because I haven't write any test case yet, but it's definitely something to test.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1384969

        SOLR-3527: SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1384969 SOLR-3527 : SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1384969

        SOLR-3527: SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1384969 SOLR-3527 : SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas.
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Mark Miller
            Reporter:
            Andy Laird
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development