Solr
  1. Solr
  2. SOLR-4321

SolrCloud collection CREATE will put multiple master shards on a single node (and no shards on some)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.1
    • Fix Version/s: 4.2, Trunk
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      This is best described by a photo of my SolrCloud admin: http://i.imgur.com/hW4xnxy.png

      I have a 4 node cluster, with shard count of 2 and replication factor of 2.

      After running something like,

      http://localhost:8983/solr/admin/collections?action=CREATE&name=15724&numShards=2&replicationFactor=2

      The shards seem to be completely randomly placed, which is fine, but some nodes receive more of the share than others (some even receiving none).

      For example, 15724 has given node "backfill-1d" 2 "parts", 15724_shard1_replica2 and 15724_shard2_replica1 but has given "backfill-2d" nothing at all.

      This creates unnecessary load on some nodes, no? Is this something I am supposed to handle myself (I looked, but don't see how).

      1. SOLR-4321.patch
        1 kB
        Eric Falcao
      2. SOLR-4321.patch
        1 kB
        Eric Falcao
      3. SOLR-4321.patch
        4 kB
        Mark Miller

        Activity

        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1438046

        SOLR-4321: Collections API will sometimes use a node more than once, even when more unused nodes are available.

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1438046 SOLR-4321 : Collections API will sometimes use a node more than once, even when more unused nodes are available.
        Hide
        Mark Miller added a comment -

        Thanks guys!

        Show
        Mark Miller added a comment - Thanks guys!
        Hide
        Commit Tag Bot added a comment -

        [trunk commit] Mark Robert Miller
        http://svn.apache.org/viewvc?view=revision&revision=1438043

        SOLR-4321: Collections API will sometimes use a node more than once, even when more unused nodes are available.

        Show
        Commit Tag Bot added a comment - [trunk commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1438043 SOLR-4321 : Collections API will sometimes use a node more than once, even when more unused nodes are available.
        Hide
        Mark Miller added a comment -

        Thanks guys - I've added a test. Will commit soon.

        Show
        Mark Miller added a comment - Thanks guys - I've added a test. Will commit soon.
        Hide
        Eric Falcao added a comment -

        Same patch but it applies at the root directory of the project

        Show
        Eric Falcao added a comment - Same patch but it applies at the root directory of the project
        Hide
        Brett Hoerner added a comment -

        Eric's patch works for me, 4 node cluster, 2 shards, 2 replicas and each node gets 1 piece of the pie.

        Show
        Brett Hoerner added a comment - Eric's patch works for me, 4 node cluster, 2 shards, 2 replicas and each node gets 1 piece of the pie.
        Show
        Eric Falcao added a comment - Here is a better permalink for brett's issue: http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201301.mbox/%3CCAC8vSzZ2hmdtm6iWc9Byf6ESSATuKc7Xq%3DoPHFNeEAWqCTtJ8Q%40mail.gmail.com%3E
        Hide
        Eric Falcao added a comment -

        I am able to confirm this using the SolrCloud example documentation found at http://wiki.apache.org/solr/SolrCloud#Example_C:_Two_shard_cluster_with_shard_replicas_and_zookeeper_ensemble

        Reproduction steps:

        • Follow example C
        • Create a new collection: curl "http://localhost:8983/solr/admin/collections?action=CREATE&name=foo&numShards=2&replicationFactor=2"
        • Verify that in the cloud admin, the new collection is imbalanced (it's not using the right mix of nodes)

        I annotated the node numbers, hoping to provide some clarity (since the ip is the same, i hope it adds some clarity):
        http://imgur.com/05L7eZ4

        Show
        Eric Falcao added a comment - I am able to confirm this using the SolrCloud example documentation found at http://wiki.apache.org/solr/SolrCloud#Example_C:_Two_shard_cluster_with_shard_replicas_and_zookeeper_ensemble Reproduction steps: Follow example C Create a new collection: curl "http://localhost:8983/solr/admin/collections?action=CREATE&name=foo&numShards=2&replicationFactor=2" Verify that in the cloud admin, the new collection is imbalanced (it's not using the right mix of nodes) I annotated the node numbers, hoping to provide some clarity (since the ip is the same, i hope it adds some clarity): http://imgur.com/05L7eZ4
        Hide
        Brett Hoerner added a comment -

        Here are some logs from the very first boot of this cluster (repeatable). "default" is in solr.xml to start, I try creating test1, test2, etc using the same curl command from above (which works fine in 4.0):

        Jan 19, 2013 11:37:05 PM org.apache.solr.core.SolrCore initIndex
        WARNING: [default] Solr index directory '/mnt/solr/data/default/index' doesn't exist. Creating new index...
        Jan 19, 2013 11:45:18 PM org.apache.solr.core.SolrCore initIndex
        WARNING: [test1_shard2_replica2] Solr index directory '/mnt/solr/data/test1_shard2_replica2/index' doesn't exist. Creating new index...
        Jan 19, 2013 11:45:19 PM org.apache.solr.common.SolrException log
        SEVERE: Sync Failed
        Jan 19, 2013 11:48:29 PM org.apache.solr.core.SolrCore initIndex
        WARNING: [test2_shard2_replica2] Solr index directory '/mnt/solr/data/test2_shard2_replica2/index' doesn't exist. Creating new index...
        Jan 19, 2013 11:48:29 PM org.apache.solr.common.SolrException log
        SEVERE: Sync Failed
        Jan 19, 2013 11:48:31 PM org.apache.solr.core.SolrCore initIndex
        WARNING: [test3_shard1_replica1] Solr index directory '/mnt/solr/data/test3_shard1_replica1/index' doesn't exist. Creating new index...
        Jan 19, 2013 11:48:32 PM org.apache.solr.common.SolrException log
        SEVERE: Sync Failed
        Jan 19, 2013 11:48:34 PM org.apache.solr.core.SolrCore initIndex
        WARNING: [test4_shard2_replica1] Solr index directory '/mnt/solr/data/test4_shard2_replica1/index' doesn't exist. Creating new index...
        Jan 19, 2013 11:48:34 PM org.apache.solr.core.SolrCore initIndex
        WARNING: [test4_shard1_replica2] Solr index directory '/mnt/solr/data/test4_shard1_replica2/index' doesn't exist. Creating new index...
        Jan 19, 2013 11:48:36 PM org.apache.solr.common.SolrException log
        SEVERE: Sync Failed
        Jan 19, 2013 11:48:37 PM org.apache.solr.core.SolrCore initIndex
        WARNING: [test5_shard2_replica2] Solr index directory '/mnt/solr/data/test5_shard2_replica2/index' doesn't exist. Creating new index...
        Jan 19, 2013 11:48:39 PM org.apache.solr.common.SolrException log
        SEVERE: Sync Failed
        Jan 19, 2013 11:48:41 PM org.apache.solr.update.PeerSync sync
        WARNING: no frame of reference to tell of we've missed updates
        Jan 19, 2013 11:50:07 PM org.apache.solr.core.SolrCore initIndex
        WARNING: [foo_shard1_replica1] Solr index directory '/mnt/solr/data/foo_shard1_replica1/index' doesn't exist. Creating new index...
        Jan 19, 2013 11:50:08 PM org.apache.solr.common.SolrException log
        SEVERE: Sync Failed

        Show
        Brett Hoerner added a comment - Here are some logs from the very first boot of this cluster (repeatable). "default" is in solr.xml to start, I try creating test1, test2, etc using the same curl command from above (which works fine in 4.0): Jan 19, 2013 11:37:05 PM org.apache.solr.core.SolrCore initIndex WARNING: [default] Solr index directory '/mnt/solr/data/default/index' doesn't exist. Creating new index... Jan 19, 2013 11:45:18 PM org.apache.solr.core.SolrCore initIndex WARNING: [test1_shard2_replica2] Solr index directory '/mnt/solr/data/test1_shard2_replica2/index' doesn't exist. Creating new index... Jan 19, 2013 11:45:19 PM org.apache.solr.common.SolrException log SEVERE: Sync Failed Jan 19, 2013 11:48:29 PM org.apache.solr.core.SolrCore initIndex WARNING: [test2_shard2_replica2] Solr index directory '/mnt/solr/data/test2_shard2_replica2/index' doesn't exist. Creating new index... Jan 19, 2013 11:48:29 PM org.apache.solr.common.SolrException log SEVERE: Sync Failed Jan 19, 2013 11:48:31 PM org.apache.solr.core.SolrCore initIndex WARNING: [test3_shard1_replica1] Solr index directory '/mnt/solr/data/test3_shard1_replica1/index' doesn't exist. Creating new index... Jan 19, 2013 11:48:32 PM org.apache.solr.common.SolrException log SEVERE: Sync Failed Jan 19, 2013 11:48:34 PM org.apache.solr.core.SolrCore initIndex WARNING: [test4_shard2_replica1] Solr index directory '/mnt/solr/data/test4_shard2_replica1/index' doesn't exist. Creating new index... Jan 19, 2013 11:48:34 PM org.apache.solr.core.SolrCore initIndex WARNING: [test4_shard1_replica2] Solr index directory '/mnt/solr/data/test4_shard1_replica2/index' doesn't exist. Creating new index... Jan 19, 2013 11:48:36 PM org.apache.solr.common.SolrException log SEVERE: Sync Failed Jan 19, 2013 11:48:37 PM org.apache.solr.core.SolrCore initIndex WARNING: [test5_shard2_replica2] Solr index directory '/mnt/solr/data/test5_shard2_replica2/index' doesn't exist. Creating new index... Jan 19, 2013 11:48:39 PM org.apache.solr.common.SolrException log SEVERE: Sync Failed Jan 19, 2013 11:48:41 PM org.apache.solr.update.PeerSync sync WARNING: no frame of reference to tell of we've missed updates Jan 19, 2013 11:50:07 PM org.apache.solr.core.SolrCore initIndex WARNING: [foo_shard1_replica1] Solr index directory '/mnt/solr/data/foo_shard1_replica1/index' doesn't exist. Creating new index... Jan 19, 2013 11:50:08 PM org.apache.solr.common.SolrException log SEVERE: Sync Failed
        Hide
        Brett Hoerner added a comment -

        I should note that the first collection created (by being in the solr.xml before the SolrCloud instances are brought up) is fine.

        I have a single collection called "default" in the config before I start my first node, and it starts with 1/2 shards and 0/2 replica (the first node). As each node is brought up for the first time it takes (in order), shard2, replica1, then replica2.

        I'd like to note I'm also having another problem that is only happening on 4.1 (not 4.0), and I'm starting to think it could be related: http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201301.mbox/browser

        Show
        Brett Hoerner added a comment - I should note that the first collection created (by being in the solr.xml before the SolrCloud instances are brought up) is fine. I have a single collection called "default" in the config before I start my first node, and it starts with 1/2 shards and 0/2 replica (the first node). As each node is brought up for the first time it takes (in order), shard2, replica1, then replica2. I'd like to note I'm also having another problem that is only happening on 4.1 (not 4.0), and I'm starting to think it could be related: http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201301.mbox/browser

          People

          • Assignee:
            Mark Miller
            Reporter:
            Brett Hoerner
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development