Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3, 4.0-ALPHA
    • Component/s: multicore
    • Labels:
      None

      Description

      There should be a provision to merge one core with another. It should be possible to create a core, add documents to it and then just merge it into the main core which is serving requests. This way, the user will not need to know the filesystem as it is needed for SOLR-1051

      1. SOLR-1331.patch
        12 kB
        Shalin Shekhar Mangar
      2. SOLR-1331.patch
        12 kB
        Shalin Shekhar Mangar
      3. SOLR-1331-branch3x.patch
        14 kB
        Shalin Shekhar Mangar

        Issue Links

          Activity

          Hide
          Hoss Man added a comment -

          Is the thinking here that we just expose SOLR-1051 type functionality at the core level w/o using file paths – in which case the caller is responsible for knowing whether or not the index merging makes logical sense; or are we talking about a more user friendly concept of core merging where we sanity check that the schemas are the same – or even more complex: merge the schemas if they don't have any conflicts (ie: all field/type names in common have identical declarations?)

          the first seems easy and essentially syntactic sugar on top of what was already implemented in SOLR-1051 (get core by name; get index dir from core; merge) but since Shalin opened a new issue for this, i suspect he's got something bigger in mind.

          Show
          Hoss Man added a comment - Is the thinking here that we just expose SOLR-1051 type functionality at the core level w/o using file paths – in which case the caller is responsible for knowing whether or not the index merging makes logical sense; or are we talking about a more user friendly concept of core merging where we sanity check that the schemas are the same – or even more complex: merge the schemas if they don't have any conflicts (ie: all field/type names in common have identical declarations?) the first seems easy and essentially syntactic sugar on top of what was already implemented in SOLR-1051 (get core by name; get index dir from core; merge) but since Shalin opened a new issue for this, i suspect he's got something bigger in mind.
          Hide
          Shalin Shekhar Mangar added a comment -

          Is the thinking here that we just expose SOLR-1051 type functionality at the core level w/o using file paths - in which case the caller is responsible for knowing whether or not the index merging makes logical sense

          Yes, that is the motivation. However, I was thinking about more safe-guards like acquiring locks on source cores to make sure a writer is not opened on them until the merge completes.

          are we talking about a more user friendly concept of core merging where we sanity check that the schemas are the same - or even more complex: merge the schemas if they don't have any conflicts (ie: all field/type names in common have identical declarations?)

          That'd be cool but also more complex. We can do that if users ask for it.

          Show
          Shalin Shekhar Mangar added a comment - Is the thinking here that we just expose SOLR-1051 type functionality at the core level w/o using file paths - in which case the caller is responsible for knowing whether or not the index merging makes logical sense Yes, that is the motivation. However, I was thinking about more safe-guards like acquiring locks on source cores to make sure a writer is not opened on them until the merge completes. are we talking about a more user friendly concept of core merging where we sanity check that the schemas are the same - or even more complex: merge the schemas if they don't have any conflicts (ie: all field/type names in common have identical declarations?) That'd be cool but also more complex. We can do that if users ask for it.
          Hide
          Yonik Seeley added a comment -

          However, I was thinking about more safe-guards like acquiring locks on source cores to make sure a writer is not opened on them until the merge completes.

          IndexWriter.addIndexes() doesn't require this does it?

          Show
          Yonik Seeley added a comment - However, I was thinking about more safe-guards like acquiring locks on source cores to make sure a writer is not opened on them until the merge completes. IndexWriter.addIndexes() doesn't require this does it?
          Hide
          Shalin Shekhar Mangar added a comment -

          IndexWriter.addIndexes() doesn't require this does it?

          It does. The javadocs for IndexWriter.addIndexesNoOptimize say:

          The index in each Directory must not be changed (opened by a writer) while this method is running. This method does not acquire a write lock in each input Directory, so it is up to the caller to enforce this.

          We can choose to leave this upto the user of the API or we can try to prevent it ourselves.

          Show
          Shalin Shekhar Mangar added a comment - IndexWriter.addIndexes() doesn't require this does it? It does. The javadocs for IndexWriter.addIndexesNoOptimize say: The index in each Directory must not be changed (opened by a writer) while this method is running. This method does not acquire a write lock in each input Directory, so it is up to the caller to enforce this. We can choose to leave this upto the user of the API or we can try to prevent it ourselves.
          Hide
          Hoss Man added a comment -

          I was thinking about more safe-guards like acquiring locks on source cores to make sure a writer is not opened on them until the merge completes.

          Ah... gotcha. yeah, it makes sense that the existing merge API (that requires you to have an index path) can get away with not worrying about this (making it the callers responsibility) but a more user-freindly api for merging by core name should probably hide this from the user.

          another option might just be to require that the (src) core is UNLOADed first.

          Show
          Hoss Man added a comment - I was thinking about more safe-guards like acquiring locks on source cores to make sure a writer is not opened on them until the merge completes. Ah... gotcha. yeah, it makes sense that the existing merge API (that requires you to have an index path) can get away with not worrying about this (making it the callers responsibility) but a more user-freindly api for merging by core name should probably hide this from the user. another option might just be to require that the (src) core is UNLOADed first.
          Hide
          Hoss Man added a comment -

          Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...

          http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E

          Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.

          A unique token for finding these 240 issues in the future: hossversioncleanup20100527

          Show
          Hoss Man added a comment - Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email... http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed. A unique token for finding these 240 issues in the future: hossversioncleanup20100527
          Hide
          Robert Muir added a comment -

          Bulk move 3.2 -> 3.3

          Show
          Robert Muir added a comment - Bulk move 3.2 -> 3.3
          Hide
          Shalin Shekhar Mangar added a comment -

          Adds a srcCore (multi-valued) parameter through which one or more source core names can be given.

          We use the IW.addIndexes(IndexReader...) method to merge the source core's indexes to the target core's index. Even if an IW is open on the source indexes, using a reader protects against corruption.

          Note - although the indexDir param also ends up calling the IW.addIndexes(IndexReader...) method, we cannot protect against open IWs on the directory so the caveat of calling commit before using mergeindexes with indexDir param still applies.

          A commit needs to be called after a merge action to see the changes.

          Show
          Shalin Shekhar Mangar added a comment - Adds a srcCore (multi-valued) parameter through which one or more source core names can be given. We use the IW.addIndexes(IndexReader...) method to merge the source core's indexes to the target core's index. Even if an IW is open on the source indexes, using a reader protects against corruption. Note - although the indexDir param also ends up calling the IW.addIndexes(IndexReader...) method, we cannot protect against open IWs on the directory so the caveat of calling commit before using mergeindexes with indexDir param still applies. A commit needs to be called after a merge action to see the changes.
          Hide
          Shalin Shekhar Mangar added a comment -

          Added a null check before calling core.close or searcher.decref

          Show
          Shalin Shekhar Mangar added a comment - Added a null check before calling core.close or searcher.decref
          Hide
          Shalin Shekhar Mangar added a comment -

          Patch ported to branch 3x

          Show
          Shalin Shekhar Mangar added a comment - Patch ported to branch 3x
          Hide
          Shalin Shekhar Mangar added a comment -

          Committed revision 1137533 on trunk and 1137555 on branch_3x

          Show
          Shalin Shekhar Mangar added a comment - Committed revision 1137533 on trunk and 1137555 on branch_3x
          Hide
          Robert Muir added a comment -

          Bulk close for 3.3

          Show
          Robert Muir added a comment - Bulk close for 3.3

            People

            • Assignee:
              Shalin Shekhar Mangar
              Reporter:
              Shalin Shekhar Mangar
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development