Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 6.0.1
    • Fix Version/s: 6.2
    • Component/s: Schema and Analysis
    • Labels:
      None
    • Environment:

      SOLR 6.0.1 running under Tomcat 8.0.35 on IBM System I

      Description

      After upgrade from 6.0.0 to 6.0.1 am getting Schema Initialization Errors.
      If I switch from ClassicSchema to Managed the Core will load first time, however subsequent loads will fail.

      Error received is :

      org.apache.solr.common.SolrException: java.lang.ExceptionInInitializerError
      at org.apache.solr.core.SolrCore.<init>(SolrCore.java:771)
      at org.apache.solr.core.SolrCore.<init>(SolrCore.java:642)
      at org.apache.solr.core.CoreContainer.create(CoreContainer.java:817)
      at org.apache.solr.core.CoreContainer.access$000(CoreContainer.java:88)
      at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:468)
      at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:459)
      at java.util.concurrent.FutureTask.run(FutureTask.java:277)
      at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
      at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$1.00000000948BC950.run(Unknown Source)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1153)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
      at java.lang.Thread.run(Thread.java:785)
      Caused by: java.lang.BootstrapMethodError: java.lang.ExceptionInInitializerError
      at org.apache.solr.schema.IndexSchema$SchemaProps$Handler.<clinit>(IndexSchema.java:1392)
      at org.apache.solr.handler.SchemaHandler.<clinit>(SchemaHandler.java:62)
      at java.lang.Class.forNameImpl(Native Method)
      at java.lang.Class.forName(Class.java:343)
      at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:530)
      at org.apache.solr.core.SolrResourceLoader.findClass(SolrResourceLoader.java:467)
      at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:565)
      at org.apache.solr.core.PluginBag.createPlugin(PluginBag.java:121)
      at org.apache.solr.core.PluginBag.init(PluginBag.java:221)
      at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:130)
      at org.apache.solr.core.SolrCore.<init>(SolrCore.java:727)
      ... 11 more
      Caused by: java.lang.ExceptionInInitializerError
      at java.lang.J9VMInternals.ensureError(J9VMInternals.java:137)
      at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:126)
      at java.lang.Class.forNameImpl(Native Method)
      at java.lang.Class.forName(Class.java:343)
      at java.lang.invoke.MethodType.nonPrimitiveClassFromString(MethodType.java:311)
      at java.lang.invoke.MethodType.parseIntoClasses(MethodType.java:373)
      at java.lang.invoke.MethodType.fromMethodDescriptorString(MethodType.java:286)
      at java.lang.invoke.MethodHandle.sendResolveMethodHandle(MethodHandle.java:961)
      at java.lang.invoke.MethodHandle.getCPMethodHandleAt(Native Method)
      at java.lang.invoke.MethodHandle.resolveInvokeDynamic(MethodHandle.java:852)
      ... 22 more
      Caused by: java.lang.NullPointerException
      at org.apache.solr.schema.IndexSchema$SchemaProps$Handler.values(IndexSchema.java:1391)
      at org.apache.solr.schema.IndexSchema$SchemaProps.<clinit>(IndexSchema.java:1503)
      ... 30 more

      1. solr.log
        70 kB
        Andrew Bennison
      2. SOLR-9179.patch
        2 kB
        Noble Paul

        Activity

        Hide
        steve_rowe Steve Rowe added a comment -

        FYI, running Solr under Tomcat is no longer supported; others attempting to reproduce will likely use the bundled Jetty.

        • How did you upgrade? Did you did a full install, or did you drop 6.0.1 components into your 6.0.0 install?
        • What do you mean by "switch from ClassicSchema to Managed"?
        • How are you triggering a core reload?
        • Can you provide your configset? Or, if not, a trimmed-down configset that can be used to reproduce the problem?
        • Were you able to perform the same steps with 6.0.0 without error?
        Show
        steve_rowe Steve Rowe added a comment - FYI, running Solr under Tomcat is no longer supported; others attempting to reproduce will likely use the bundled Jetty. How did you upgrade? Did you did a full install, or did you drop 6.0.1 components into your 6.0.0 install? What do you mean by "switch from ClassicSchema to Managed"? How are you triggering a core reload? Can you provide your configset? Or, if not, a trimmed-down configset that can be used to reproduce the problem? Were you able to perform the same steps with 6.0.0 without error?
        Hide
        andrewbennison Andrew Bennison added a comment -

        I replaced contrib and dist directories with 6.0.1 versions and replaced webapp with 6.0.1 version.
        Updated solrconfig to reflect new version ie lucene version from 6.0.0 to 6.0.1
        We use ClassicSchema on current 5.x Production and 6.0.0 Development Versions

        With Schema set at classic get the initialization Error as soon as I start everything up.

        If I replace <schemaFactory class="ClassicIndexSchemaFactory"/> with

        <schemaFactory class="ManagedIndexSchemaFactory">
        <bool name="mutable">true</bool>
        <str name="managedSchemaResourceName">managed-schema</str>
        </schemaFactory>

        Then when I load it creates the managed-schema and loads the core.

        If I then shutdown and restart I get the Error.

        If I rename schema.xml.bak to schema.xml and delete managed-schema it will load again OK.

        everything is fine running under 6.0.0

        Show
        andrewbennison Andrew Bennison added a comment - I replaced contrib and dist directories with 6.0.1 versions and replaced webapp with 6.0.1 version. Updated solrconfig to reflect new version ie lucene version from 6.0.0 to 6.0.1 We use ClassicSchema on current 5.x Production and 6.0.0 Development Versions With Schema set at classic get the initialization Error as soon as I start everything up. If I replace <schemaFactory class="ClassicIndexSchemaFactory"/> with <schemaFactory class="ManagedIndexSchemaFactory"> <bool name="mutable">true</bool> <str name="managedSchemaResourceName">managed-schema</str> </schemaFactory> Then when I load it creates the managed-schema and loads the core. If I then shutdown and restart I get the Error. If I rename schema.xml.bak to schema.xml and delete managed-schema it will load again OK. everything is fine running under 6.0.0
        Hide
        andrewbennison Andrew Bennison added a comment -

        As far as Jetty is concerned, it needs a lot of work to get it to run on an IBM System I, I can try that but it will take me a week or two, so far running under tomcat has had few issues.

        Show
        andrewbennison Andrew Bennison added a comment - As far as Jetty is concerned, it needs a lot of work to get it to run on an IBM System I, I can try that but it will take me a week or two, so far running under tomcat has had few issues.
        Hide
        steve_rowe Steve Rowe added a comment -

        I suggest you try doing a full 6.0.1 install, then copy in your custom configs etc., rather than the partial overwrite thing you did.

        With Schema set at classic get the initialization Error as soon as I start everything up.

        So you're getting this initialization error regardless of whether it's classic or managed, right? (It's just that under managed, it only shows up after reload.)

        Show
        steve_rowe Steve Rowe added a comment - I suggest you try doing a full 6.0.1 install, then copy in your custom configs etc., rather than the partial overwrite thing you did. With Schema set at classic get the initialization Error as soon as I start everything up. So you're getting this initialization error regardless of whether it's classic or managed, right? (It's just that under managed, it only shows up after reload.)
        Hide
        andrewbennison Andrew Bennison added a comment -

        Yes with Classic it fails at startup.

        With managed it does the conversion , loads the core and is OK until reloaded.

        Show
        andrewbennison Andrew Bennison added a comment - Yes with Classic it fails at startup. With managed it does the conversion , loads the core and is OK until reloaded.
        Hide
        andrewbennison Andrew Bennison added a comment -

        Ran config under Jetty on Windows OK
        Ported to the IBM system I and it then failed (see attached log)

        Version of Java on Windows

        java version "1.8.0_91"
        Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
        Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

        Version of Java on the IBM Power System I

        java version "1.8.0"
        Java(TM) SE Runtime Environment (build pap6480sr2fp10-20160108_01(SR2 FP10))
        IBM J9 VM (build 2.8, JRE 1.8.0 OS/400 ppc64-64 Compressed References jvmap6480sr2fp10-20160108_01_cr (JIT enabled, AOT enabled)
        J9VM - R28_20160106_1341_B284759
        JIT - tr.r14.java_20151209_107110.02
        GC - R28_20160106_1341_B284759_CMPRSS
        J9CL - 20160106_284759)
        JCL - 20151231_01 based on Oracle jdk8u71-b15

        Show
        andrewbennison Andrew Bennison added a comment - Ran config under Jetty on Windows OK Ported to the IBM system I and it then failed (see attached log) Version of Java on Windows java version "1.8.0_91" Java(TM) SE Runtime Environment (build 1.8.0_91-b14) Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode) Version of Java on the IBM Power System I java version "1.8.0" Java(TM) SE Runtime Environment (build pap6480sr2fp10-20160108_01(SR2 FP10)) IBM J9 VM (build 2.8, JRE 1.8.0 OS/400 ppc64-64 Compressed References jvmap6480sr2fp10-20160108_01_cr (JIT enabled, AOT enabled) J9VM - R28_20160106_1341_B284759 JIT - tr.r14.java_20151209_107110.02 GC - R28_20160106_1341_B284759_CMPRSS J9CL - 20160106_284759) JCL - 20151231_01 based on Oracle jdk8u71-b15
        Hide
        Mobe Moritz Becker added a comment - - edited

        I am facing the same issue on the latest J9, switched from Solr 5.3.1 to Sorl 6.0.1.

        IBM J9 VM (build 2.8, JRE 1.8.0 AIX ppc64-64 Compressed References 20160427_301573 (JIT enabled, AOT enabled)
        J9VM - R28_Java8_SR3_20160427_1620_B301573
        JIT - tr.r14.java.green_20160329_114288
        GC - R28_Java8_SR3_20160427_1620_B301573_CMPRSS
        J9CL - 20160427_301573)
        JCL - 20160421_01 based on Oracle jdk8u91-b14

        I am using the out-of-the-box solr installation.

        Show
        Mobe Moritz Becker added a comment - - edited I am facing the same issue on the latest J9, switched from Solr 5.3.1 to Sorl 6.0.1. IBM J9 VM (build 2.8, JRE 1.8.0 AIX ppc64-64 Compressed References 20160427_301573 (JIT enabled, AOT enabled) J9VM - R28_Java8_SR3_20160427_1620_B301573 JIT - tr.r14.java.green_20160329_114288 GC - R28_Java8_SR3_20160427_1620_B301573_CMPRSS J9CL - 20160427_301573) JCL - 20160421_01 based on Oracle jdk8u91-b14 I am using the out-of-the-box solr installation.
        Hide
        cjcowie Colvin Cowie added a comment - - edited

        We too have this problem, out of the box on Solr 6.1.0 on Windows, it fails even on the basic_configs cloud example.
        Solr 6.0 is fine.

        java version "1.8.0"
        Java(TM) SE Runtime Environment (build pwa6480sr2-20151023_01(SR2))
        IBM J9 VM (build 2.8, JRE 1.8.0 Windows 7 amd64-64 Compressed References 20151019_272764 (JIT enabled, AOT enabled)
        J9VM - R28_Java8_SR2_20151019_2144_B272764
        JIT - tr.r14.java_20151006_102517.04
        GC - R28_Java8_SR2_20151019_2144_B272764_CMPRSS
        J9CL - 20151019_272764)
        JCL - 20151022_01 based on Oracle jdk8u65-b17

        I believe Noble Paul added the org.apache.solr.schema.IndexSchema$SchemaProps$Handler.

        Show
        cjcowie Colvin Cowie added a comment - - edited We too have this problem, out of the box on Solr 6.1.0 on Windows, it fails even on the basic_configs cloud example. Solr 6.0 is fine. java version "1.8.0" Java(TM) SE Runtime Environment (build pwa6480sr2-20151023_01(SR2)) IBM J9 VM (build 2.8, JRE 1.8.0 Windows 7 amd64-64 Compressed References 20151019_272764 (JIT enabled, AOT enabled) J9VM - R28_Java8_SR2_20151019_2144_B272764 JIT - tr.r14.java_20151006_102517.04 GC - R28_Java8_SR2_20151019_2144_B272764_CMPRSS J9CL - 20151019_272764) JCL - 20151022_01 based on Oracle jdk8u65-b17 I believe Noble Paul added the org.apache.solr.schema.IndexSchema$SchemaProps$Handler.
        Hide
        noble.paul Noble Paul added a comment -

        Can you just give me the steps to reproduce this.

        Show
        noble.paul Noble Paul added a comment - Can you just give me the steps to reproduce this.
        Hide
        cjcowie Colvin Cowie added a comment - - edited

        Do you have an IBM JDK? You should be able to get one from https://www.ibm.com/developerworks/java/jdk/

        To reproduce, just set your JAVA_HOME for the IBM java, and run solr -e cloud -noprompt and then it will fail on the creation of the collection.

        Creating new collection 'gettingstarted' using command: http://localhost:8983/solr/admin/collections?action=CREATE&name=gettingstarted&numShards=2&replicationFactor=2&maxShardsPerNode=2&collection.configName=gettingstarted

        ERROR: Failed to create collection 'gettingstarted' due to: {192.168.56.7:8983_solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://192.168.56.7:8983/solr: Error CREATEing SolrCore 'gettingstarted_shard1_replica1': Unable to create core [gettingstarted_shard1_replica1] Caused by:null, 192.168.56.7:7574_solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://192.168.56.7:7574/solr: Error CREATEing SolrCore 'gettingstarted_shard1_replica2': Unable to create core [gettingstarted_shard1_replica2] Caused by: null}

        ERROR: Failed to create collection using command: [-name, gettingstarted, -shards, 2, -replicationFactor, 2, -confname, gettingstarted, -confdir, data_driven_schema_configs, -configsetsDir, C:\solr\solr-6.1.0\solr-6.1.0\server\solr\configsets, -solrUrl, http://localhost:8983/solr]

        Show
        cjcowie Colvin Cowie added a comment - - edited Do you have an IBM JDK? You should be able to get one from https://www.ibm.com/developerworks/java/jdk/ To reproduce, just set your JAVA_HOME for the IBM java, and run solr -e cloud -noprompt and then it will fail on the creation of the collection. Creating new collection 'gettingstarted' using command: http://localhost:8983/solr/admin/collections?action=CREATE&name=gettingstarted&numShards=2&replicationFactor=2&maxShardsPerNode=2&collection.configName=gettingstarted ERROR: Failed to create collection 'gettingstarted' due to: {192.168.56.7:8983_solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://192.168.56.7:8983/solr: Error CREATEing SolrCore 'gettingstarted_shard1_replica1': Unable to create core [gettingstarted_shard1_replica1] Caused by:null , 192.168.56.7:7574_solr=org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:Error from server at http://192.168.56.7:7574/solr: Error CREATEing SolrCore 'gettingstarted_shard1_replica2': Unable to create core [gettingstarted_shard1_replica2] Caused by: null } ERROR: Failed to create collection using command: [-name, gettingstarted, -shards, 2, -replicationFactor, 2, -confname, gettingstarted, -confdir, data_driven_schema_configs, -configsetsDir, C:\solr\solr-6.1.0\solr-6.1.0\server\solr\configsets, -solrUrl, http://localhost:8983/solr]
        Hide
        noble.paul Noble Paul added a comment -

        Do you mean it is not reproducible in SUN JDK ?

        Show
        noble.paul Noble Paul added a comment - Do you mean it is not reproducible in SUN JDK ?
        Hide
        cjcowie Colvin Cowie added a comment -

        Yes. It appears to be fine in the Oracle JVM. I'm planning to raise a PMR on the IBM JVM, but no idea what the timescale for it being resolved will be.

        Show
        cjcowie Colvin Cowie added a comment - Yes. It appears to be fine in the Oracle JVM. I'm planning to raise a PMR on the IBM JVM, but no idea what the timescale for it being resolved will be.
        Hide
        cjcowie Colvin Cowie added a comment -

        So, I've had a look what is happening.

        It appears that in the IBM JVM the assignment of nameMapping in SchemaProps happens before the Handler enum values have been constructed, while in the Oracle JVM the assignment only happens when nameMapping is actually referenced from SchemaHandler#handleGET. The assignment of nameMapping results in a call to Handler.values() which blows up since the values aren't constructed yet.

        The premature assignment of nameMapping appears to happen as a result of the functions being from SchemaProps -> Object, which leads to the IBM JVM trying to initialize the static fields on SchemaProps as soon as it hits sp -> sp.schema.getSchemaName().

        Moving nameMapping out of SchemaProps and into IndexSchema itself 'fixes' the problem it seems.

        I'll raise a PMR on the IBM JVM and see what happens.

        Show
        cjcowie Colvin Cowie added a comment - So, I've had a look what is happening. It appears that in the IBM JVM the assignment of nameMapping in SchemaProps happens before the Handler enum values have been constructed, while in the Oracle JVM the assignment only happens when nameMapping is actually referenced from SchemaHandler#handleGET . The assignment of nameMapping results in a call to Handler.values() which blows up since the values aren't constructed yet. The premature assignment of nameMapping appears to happen as a result of the functions being from SchemaProps -> Object, which leads to the IBM JVM trying to initialize the static fields on SchemaProps as soon as it hits sp -> sp.schema.getSchemaName() . Moving nameMapping out of SchemaProps and into IndexSchema itself 'fixes' the problem it seems. I'll raise a PMR on the IBM JVM and see what happens.
        Hide
        noble.paul Noble Paul added a comment -

        Post a patch and let's fix it anyway

        Show
        noble.paul Noble Paul added a comment - Post a patch and let's fix it anyway
        Hide
        cjcowie Colvin Cowie added a comment -

        Unfortunately I'm not (currently) able to make actual code contributions. I'm going to see about getting approval to do so. Hopefully it won't take that long

        Show
        cjcowie Colvin Cowie added a comment - Unfortunately I'm not (currently) able to make actual code contributions. I'm going to see about getting approval to do so. Hopefully it won't take that long
        Hide
        erickerickson Erick Erickson added a comment -

        Is it OK for you to just add a comment like "move line XXX from file YYY to line WWW of file ZZZ"? Or are there too many to deal with so simply?

        Show
        erickerickson Erick Erickson added a comment - Is it OK for you to just add a comment like "move line XXX from file YYY to line WWW of file ZZZ"? Or are there too many to deal with so simply?
        Hide
        cjcowie Colvin Cowie added a comment - - edited

        So, in org.apache.solr.schema.IndexSchema it's a case of moving line 1503 below 1505. i.e. move nameMapping out of SchemaProps and into IndexSchema itself.

        The other 2 changes are just to just to fix the knock on effects, which Eclipse will show up readily - update the references to Handler in the nameMapping definition and the reference to nameMapping in org.apache.solr.schema.IndexSchema.SchemaProps.

        Show
        cjcowie Colvin Cowie added a comment - - edited So, in org.apache.solr.schema.IndexSchema it's a case of moving line 1503 below 1505. i.e. move nameMapping out of SchemaProps and into IndexSchema itself. The other 2 changes are just to just to fix the knock on effects, which Eclipse will show up readily - update the references to Handler in the nameMapping definition and the reference to nameMapping in org.apache.solr.schema.IndexSchema.SchemaProps .
        Hide
        noble.paul Noble Paul added a comment -

        Colvin Cowie Is this correct?

        Show
        noble.paul Noble Paul added a comment - Colvin Cowie Is this correct?
        Hide
        cjcowie Colvin Cowie added a comment -

        Yeah, that looks good.

        Show
        cjcowie Colvin Cowie added a comment - Yeah, that looks good.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 17797673f2eb152c09af022e5d2e03bf5c1d584b in lucene-solr's branch refs/heads/master from Noble Paul
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1779767 ]

        SOLR-9179: NPE in IndexSchema using IBM JDK

        Show
        jira-bot ASF subversion and git services added a comment - Commit 17797673f2eb152c09af022e5d2e03bf5c1d584b in lucene-solr's branch refs/heads/master from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1779767 ] SOLR-9179 : NPE in IndexSchema using IBM JDK
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 750bb367d26d0b536c24a75f1aa88582cb27517d in lucene-solr's branch refs/heads/branch_6x from Noble Paul
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=750bb36 ]

        SOLR-9179: NPE in IndexSchema using IBM JDK

        Show
        jira-bot ASF subversion and git services added a comment - Commit 750bb367d26d0b536c24a75f1aa88582cb27517d in lucene-solr's branch refs/heads/branch_6x from Noble Paul [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=750bb36 ] SOLR-9179 : NPE in IndexSchema using IBM JDK
        Hide
        cjcowie Colvin Cowie added a comment - - edited

        A fix for the IBM JVM error will be available in the IBM JRE 8.0.3.20 - http://www-01.ibm.com/support/docview.wss?uid=swg1IV88715

        Show
        cjcowie Colvin Cowie added a comment - - edited A fix for the IBM JVM error will be available in the IBM JRE 8.0.3.20 - http://www-01.ibm.com/support/docview.wss?uid=swg1IV88715

          People

          • Assignee:
            noble.paul Noble Paul
            Reporter:
            andrewbennison Andrew Bennison
          • Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:

              Development