Solr
  1. Solr
  2. SOLR-6643

Core load silently aborted if missing schema has depenencies - LinkageErrors swollowed

    Details

      Description

      How to reproduce

      1. Start with standard collection1 config
      2. Add a field type to schema using the ICU contrib, no need for a field
        <fieldType name="text_icu" class="solr.TextField">
          <analyzer><tokenizer class="solr.ICUTokenizerFactory"/></analyzer>
        </fieldType>
        
      3. cd example
      4. mkdir solr/lib
      5. cp ../contrib/analysis-extras/lucene-libs/lucene-analyzers-icu-4.10.1.jar solr/lib/
      6. bin/solr -f
      7. Core is not loaded, and no messages in log after this line
        ... INFO  org.apache.solr.schema.IndexSchema  – [collection1] Schema name=example
        

      Note that we did not add the dependency libs from analysis-extras/lib, so we'd expect a ClassNotFoundException, but some way the initialization of schema aborts silently. The ICUTokenizerFactory is instansiated by reflection and I suspect that some exception is swallowed in AbstractPluginLoader#create()

      1. SOLR-6643.patch
        16 kB
        Hoss Man
      2. SOLR-6643.patch
        16 kB
        Hoss Man

        Activity

        Hide
        Jan Høydahl added a comment -

        Any ideas for how to solve this?

        Show
        Jan Høydahl added a comment - Any ideas for how to solve this?
        Hide
        Hoss Man added a comment -

        I was able to reproduce Jan's steps, and then while trying again using core CREATE instead of relying on startup, was able to get an error returned to the client indicating the root cause of the problem...

        java.lang.NoClassDefFoundError: com/ibm/icu/text/BreakIterator
        	at java.lang.Class.getDeclaredConstructors0(Native Method)
        	at java.lang.Class.privateGetDeclaredConstructors(Class.java:2663)
        	at java.lang.Class.getConstructor0(Class.java:3067)
        	at java.lang.Class.getConstructor(Class.java:1817)
        	at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:604)
        	at org.apache.solr.schema.FieldTypePluginLoader$2.create(FieldTypePluginLoader.java:333)
        	at org.apache.solr.schema.FieldTypePluginLoader$2.create(FieldTypePluginLoader.java:326)
        	at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:151)
        

        ...which kind of makes sense: nothing in the plugin architetcure does much in the way of "catching" LinkageError's like NoClassDefFoundError because in general trying ot catch Error's is usually a bad idea.

        i think however that it would make sense for SolrResourceLoader to at least catch+log+rethrow any error from newInstance

        that still leaves a second question however: the NoClassDefFoundError should (currently) be bubbling all the way up to the CoreContainer which should then be recording it as an init failure for the core – but that isn't happening. why?

        based on my testing, if a java.lang.Error is thrown during init of something like RequestHandler, then that gets properly tracked as a coreInitFailure – but if something from the schema (ie: FieldType) throws a j.l.Error, that gets swallowed up.

        Show
        Hoss Man added a comment - I was able to reproduce Jan's steps, and then while trying again using core CREATE instead of relying on startup, was able to get an error returned to the client indicating the root cause of the problem... java.lang.NoClassDefFoundError: com/ibm/icu/text/BreakIterator at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2663) at java.lang.Class.getConstructor0(Class.java:3067) at java.lang.Class.getConstructor(Class.java:1817) at org.apache.solr.core.SolrResourceLoader.newInstance(SolrResourceLoader.java:604) at org.apache.solr.schema.FieldTypePluginLoader$2.create(FieldTypePluginLoader.java:333) at org.apache.solr.schema.FieldTypePluginLoader$2.create(FieldTypePluginLoader.java:326) at org.apache.solr.util.plugin.AbstractPluginLoader.load(AbstractPluginLoader.java:151) ...which kind of makes sense: nothing in the plugin architetcure does much in the way of "catching" LinkageError's like NoClassDefFoundError because in general trying ot catch Error's is usually a bad idea. i think however that it would make sense for SolrResourceLoader to at least catch+log+rethrow any error from newInstance that still leaves a second question however: the NoClassDefFoundError should (currently) be bubbling all the way up to the CoreContainer which should then be recording it as an init failure for the core – but that isn't happening. why? based on my testing, if a java.lang.Error is thrown during init of something like RequestHandler, then that gets properly tracked as a coreInitFailure – but if something from the schema (ie: FieldType) throws a j.l.Error, that gets swallowed up.
        Hide
        Hoss Man added a comment -

        Here's a patch that does a few things:

        • adds 2 tests to CoreContainerCoreInitFailuresTest
          • testJavaLangErrorFromHandlerOnStartup demonstrates that a new ThrowErrorOnInitRequestHandler causes a correctly tracked coreInitFailure
          • testJavaLangErrorFromSchemaOnStartup demonstrates that a new ThrowErrorOnInitFieldType causes a correctly tracked coreInitFailure
            • this test currently fails
        • SolrResourceLoader...
          • now logs & re-throws any execptions
          • has some refactoring to remove copy/paste code
            • nocommits noted here until more agressive testing is done.

        I still don't understand why testJavaLangErrorFromSchemaOnStartup, and i won't have much time to work on it in the near future – but it may be a good idea to commit this patch as is with an @Ignore on that test linking to a new spin-off issue – so that we can at least get te SolrResourceLoader logging in place for people who run into problems like this .

        Show
        Hoss Man added a comment - Here's a patch that does a few things: adds 2 tests to CoreContainerCoreInitFailuresTest testJavaLangErrorFromHandlerOnStartup demonstrates that a new ThrowErrorOnInitRequestHandler causes a correctly tracked coreInitFailure testJavaLangErrorFromSchemaOnStartup demonstrates that a new ThrowErrorOnInitFieldType causes a correctly tracked coreInitFailure this test currently fails SolrResourceLoader... now logs & re-throws any execptions has some refactoring to remove copy/paste code nocommits noted here until more agressive testing is done. I still don't understand why testJavaLangErrorFromSchemaOnStartup, and i won't have much time to work on it in the near future – but it may be a good idea to commit this patch as is with an @Ignore on that test linking to a new spin-off issue – so that we can at least get te SolrResourceLoader logging in place for people who run into problems like this .
        Hide
        Alan Woodward added a comment -

        but that isn't happening. why?

        I think it's because CoreContainer.create() is catching Exceptions, not Errors, when updating the coreInitFailure map. Maybe the best solution is for SolrResourceLoader to try and catch LinkageErrors and rethrow it as a SolrException. Catching classloader problems is I think within the resource loader's remit (unlike out of memory errors, etc).

        Show
        Alan Woodward added a comment - but that isn't happening. why? I think it's because CoreContainer.create() is catching Exceptions, not Errors, when updating the coreInitFailure map. Maybe the best solution is for SolrResourceLoader to try and catch LinkageErrors and rethrow it as a SolrException. Catching classloader problems is I think within the resource loader's remit (unlike out of memory errors, etc).
        Hide
        Hoss Man added a comment -

        I think it's because CoreContainer.create() is catching Exceptions, not Errors, when updating the coreInitFailure map.

        Sure - but the piece i couldn't make sense of is why/where testJavaLangErrorFromHandlerOnStartup passed (and those Errors ere in coreInitFailures) but testJavaLangErrorFromSchemaOnStartup didn't.

        digging arround a bit more, it looks like this is because SolrCore() is wrapping some types of Throwable in SolrException (shudder) but the IndexSchema already exists before the SolreCore constructor is called, and any Errors that come from it don't get similar wrapping.

        Maybe the best solution is for SolrResourceLoader to try and catch LinkageErrors and rethrow it as a SolrException. Catching classloader problems is I think within the resource loader's remit (unlike out of memory errors, etc).

        maybe - but that's a slippery slope i'd rather avoid – i'm catching & re-throwing Errors is one thing, wrapping Errors in Exceptions is something i'm very much not a fan of.

        i think a safer (and more all encompasing) fix would be for CoreContainer to handle wraping Errors in SolrException - not for the purpose of re-throwing, but just for tracking in coreInitFailures. that way even for things like OOM or IOError during core init, we still have a note about it in coreInitFailures.

        Attaching an updated patch that goes this direction - still running tests, but review/comments appreciated

        Show
        Hoss Man added a comment - I think it's because CoreContainer.create() is catching Exceptions, not Errors, when updating the coreInitFailure map. Sure - but the piece i couldn't make sense of is why/where testJavaLangErrorFromHandlerOnStartup passed (and those Errors ere in coreInitFailures) but testJavaLangErrorFromSchemaOnStartup didn't. digging arround a bit more, it looks like this is because SolrCore() is wrapping some types of Throwable in SolrException (shudder) but the IndexSchema already exists before the SolreCore constructor is called, and any Errors that come from it don't get similar wrapping. Maybe the best solution is for SolrResourceLoader to try and catch LinkageErrors and rethrow it as a SolrException. Catching classloader problems is I think within the resource loader's remit (unlike out of memory errors, etc). maybe - but that's a slippery slope i'd rather avoid – i'm catching & re-throwing Errors is one thing, wrapping Errors in Exceptions is something i'm very much not a fan of. i think a safer (and more all encompasing) fix would be for CoreContainer to handle wraping Errors in SolrException - not for the purpose of re-throwing, but just for tracking in coreInitFailures. that way even for things like OOM or IOError during core init, we still have a note about it in coreInitFailures. — Attaching an updated patch that goes this direction - still running tests, but review/comments appreciated
        Hide
        ASF subversion and git services added a comment -

        Commit 1650350 from hossman@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1650350 ]

        SOLR-6643: Fix error reporting & logging of low level JVM Errors that occur when loading/reloading a SolrCore

        Show
        ASF subversion and git services added a comment - Commit 1650350 from hossman@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1650350 ] SOLR-6643 : Fix error reporting & logging of low level JVM Errors that occur when loading/reloading a SolrCore
        Hide
        ASF subversion and git services added a comment -

        Commit 1650367 from hossman@apache.org in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1650367 ]

        SOLR-6643: Fix error reporting & logging of low level JVM Errors that occur when loading/reloading a SolrCore (merge r1650350)

        Show
        ASF subversion and git services added a comment - Commit 1650367 from hossman@apache.org in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1650367 ] SOLR-6643 : Fix error reporting & logging of low level JVM Errors that occur when loading/reloading a SolrCore (merge r1650350)
        Hide
        Anshum Gupta added a comment -

        Bulk close after 5.0 release.

        Show
        Anshum Gupta added a comment - Bulk close after 5.0 release.

          People

          • Assignee:
            Hoss Man
            Reporter:
            Jan Høydahl
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development