Solr
  1. Solr
  2. SOLR-3527

SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas.

    Details

      Description

      Send the following command to a Solr server with many segments in a multi-shard, multi-server environment:

      curl "http://localhost:8080/solr/update?optimize=true&waitFlush=true&maxSegments=6&distrib=false"

      The local server will end up with the number of segments at 6, as requested, but all other shards in the index will be optimized with maxSegments=1, which takes far longer to complete. All shards should be optimized to the requested value of 6.

        Activity

        Andy Laird created issue -
        Hide
        Andy Laird added a comment -

        One additional data point: the distrib=false does not matter with current behavior. It seems if distrib=false only the local server should be optimized (to the requested value) and if distrib=true (default) all shards in the index should be optimized with N max segments.

        Show
        Andy Laird added a comment - One additional data point: the distrib=false does not matter with current behavior. It seems if distrib=false only the local server should be optimized (to the requested value) and if distrib=true (default) all shards in the index should be optimized with N max segments.
        Hide
        Mark Miller added a comment -

        Sounds right Andy - thanks for the report.

        Show
        Mark Miller added a comment - Sounds right Andy - thanks for the report.
        Hide
        Tomás Fernández Löbbe added a comment - - edited

        I was investigating this issue and found that DistribUpdateProcessor distributes the commit commands using the SolrCommandDistributor.distribCommit(...). The method creates Actions to be distributed to other nodes like:

        void addCommit(UpdateRequestExt ureq, CommitUpdateCommand cmd) {
            if (cmd == null) return;
            ureq.setAction(cmd.optimize ? AbstractUpdateRequest.ACTION.OPTIMIZE
                : AbstractUpdateRequest.ACTION.COMMIT, false, cmd.waitSearcher);
          }
        

        In that method, the action is not considering the "maxOptimizeSegments" parameter, that's why it's not being distributed to other nodes. However, a bigger problem may be that this method is omitting other parameters too, like "softCommit" and "epungeDeletes", which means that an explicit soft commit issued like:

        http://host:port/solr/update?commit=true&softCommit=true
        

        can be distributed as a hard commit to the nodes. I'm not sure about this because I haven't write any test case yet, but it's definitely something to test.

        Show
        Tomás Fernández Löbbe added a comment - - edited I was investigating this issue and found that DistribUpdateProcessor distributes the commit commands using the SolrCommandDistributor.distribCommit(...). The method creates Actions to be distributed to other nodes like: void addCommit(UpdateRequestExt ureq, CommitUpdateCommand cmd) { if (cmd == null ) return ; ureq.setAction(cmd.optimize ? AbstractUpdateRequest.ACTION.OPTIMIZE : AbstractUpdateRequest.ACTION.COMMIT, false , cmd.waitSearcher); } In that method, the action is not considering the "maxOptimizeSegments" parameter, that's why it's not being distributed to other nodes. However, a bigger problem may be that this method is omitting other parameters too, like "softCommit" and "epungeDeletes", which means that an explicit soft commit issued like: http: //host:port/solr/update?commit= true &softCommit= true can be distributed as a hard commit to the nodes. I'm not sure about this because I haven't write any test case yet, but it's definitely something to test.
        Mark Miller made changes -
        Field Original Value New Value
        Assignee Mark Miller [ markrmiller@gmail.com ]
        Mark Miller made changes -
        Fix Version/s 4.0 [ 12322551 ]
        Fix Version/s 5.0 [ 12321664 ]
        Priority Major [ 3 ] Critical [ 2 ]
        Component/s SolrCloud [ 12313841 ]
        Mark Miller made changes -
        Summary Optimize ignores maxSegments in distributed environment SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas.
        Mark Miller made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1384969

        SOLR-3527: SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1384969 SOLR-3527 : SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1384969

        SOLR-3527: SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1384969 SOLR-3527 : SolrCmdDistributor drops some of the important commit attributes (maxOptimizeSegments, softCommit, expungeDeletes) when sending a commit to replicas.
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.
        Uwe Schindler made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        98d 6h 4m 1 Mark Miller 14/Sep/12 23:00
        Resolved Resolved Closed Closed
        237d 11h 33m 1 Uwe Schindler 10/May/13 10:34

          People

          • Assignee:
            Mark Miller
            Reporter:
            Andy Laird
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development