Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: 16.11.01
    • Component/s: solr

      Description

      Apache Solr, a modern search engine, seems to be one of the most commonly requested feature within the OFBiz community.

      The original text of the description can be found under the attachments.

      1. solr-v1.00-r4284.zip
        7.55 MB
        Paul Piper
      2. solr in specialpurpose splitted.zip
        6.39 MB
        Jacques Le Roux
      3. solr in specialpurpose splitted.z02
        9.00 MB
        Jacques Le Roux
      4. solr in specialpurpose splitted.z01
        9.00 MB
        Jacques Le Roux
      5. solr in specialpurpose.zip
        7.65 MB
        Jacques Le Roux
      6. ofbiz-solr-logo-screenshot.jpg
        90 kB
        Shi Jinghai
      7. ofbiz-solr-logo.png
        8 kB
        Shi Jinghai
      8. OFBIZ-5042 - solr in specialpurpose.patch
        4 kB
        Jacques Le Roux
      9. OFBIZ-5042 - solr in specialpurpose.patch
        5 kB
        Jacques Le Roux
      10. OFBIZ-5042 - solr in specialpurpose.patch
        5 kB
        Jacques Le Roux
      11. OFBIZ-5042-OriginalDescription.txt
        10 kB
        Pierre Smits
      12. OFBIZ-5042-OFBizComponent.patch
        0.7 kB
        Pierre Smits
      13. 2013-07-30 11 52 52.png
        21 kB
        Jacques Le Roux

        Issue Links

          Activity

          Hide
          madppiper Paul Piper added a comment -

          Attached you can find the standalone solr implementation as created by Pascal Proulx last night.

          Show
          madppiper Paul Piper added a comment - Attached you can find the standalone solr implementation as created by Pascal Proulx last night.
          Hide
          djjava Tri Duc Vo added a comment -

          Nice feature
          Cheers

          Show
          djjava Tri Duc Vo added a comment - Nice feature Cheers
          Hide
          brettonwoods Tom Burns added a comment -

          Hi Paul,

          I was thinking of using Solar / Tika for processing resumes in the HR application. Your work is a great help in jump starting what I have in mind. Thanks!

          I ran into one problem. I work in Eclipse and invoke OFBiz using Eclipse Dbug with the following configuration.

          -Xms128M -Xmx512M -XX:MaxPermSize=512M
          -Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false

          When I added -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar to the configuration it throws:

          throw (StartupException) new StartupException("Couldn't not fetch config instance").initCause(e);

          I altered the root build xml as described above and was able to run in Eclipse by using the ant start and stop commands but I would like to run in a debug configuration.

          Do you have a VM Arguments command for Eclipse that starts OFBiz with Solar?

          Tom

          Show
          brettonwoods Tom Burns added a comment - Hi Paul, I was thinking of using Solar / Tika for processing resumes in the HR application. Your work is a great help in jump starting what I have in mind. Thanks! I ran into one problem. I work in Eclipse and invoke OFBiz using Eclipse Dbug with the following configuration. -Xms128M -Xmx512M -XX:MaxPermSize=512M -Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false When I added -Dsolr.solr.home=hot-deploy/solr -jar ofbiz.jar to the configuration it throws: throw (StartupException) new StartupException("Couldn't not fetch config instance").initCause(e); I altered the root build xml as described above and was able to run in Eclipse by using the ant start and stop commands but I would like to run in a debug configuration. Do you have a VM Arguments command for Eclipse that starts OFBiz with Solar? Tom
          Hide
          holivier Olivier Heintz added a comment -

          Thank you very much for the very clear documentation and process to test.
          One more things in my todo.
          Cheers.

          Show
          holivier Olivier Heintz added a comment - Thank you very much for the very clear documentation and process to test. One more things in my todo. Cheers.
          Hide
          pproulx Pascal Proulx added a comment -

          @Tom Burns:

          We got Eclipse debugging to work based on the exact instructions detailed here:
          https://cwiki.apache.org/OFBIZ/running-and-debugging-ofbiz-in-eclipse.html#RunningandDebuggingOFBizinEclipse-debuggingInEclipse

          Afterward, we simply add the line "-Dsolr.solr.home=hot-deploy/solr" to the VM arguments line under the Arguments tab (following -Xms...), but without the "-jar ofbiz.jar" argument (since you already add it to the classpath under User Entries and then specify main as org.ofbiz.base.start.Start). The -jar is only for startofbiz.sh/bat.

          So this command worked when attempted:
          -Xms128M -Xmx512M -XX:MaxPermSize=512M
          -Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dsolr.solr.home=hot-deploy/solr

          Hope this helps

          Show
          pproulx Pascal Proulx added a comment - @Tom Burns: We got Eclipse debugging to work based on the exact instructions detailed here: https://cwiki.apache.org/OFBIZ/running-and-debugging-ofbiz-in-eclipse.html#RunningandDebuggingOFBizinEclipse-debuggingInEclipse Afterward, we simply add the line "-Dsolr.solr.home=hot-deploy/solr" to the VM arguments line under the Arguments tab (following -Xms...), but without the "-jar ofbiz.jar" argument (since you already add it to the classpath under User Entries and then specify main as org.ofbiz.base.start.Start). The -jar is only for startofbiz.sh/bat. So this command worked when attempted: -Xms128M -Xmx512M -XX:MaxPermSize=512M -Dcom.sun.management.jmxremote.port=3333 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dsolr.solr.home=hot-deploy/solr Hope this helps
          Hide
          brettonwoods Tom Burns added a comment -

          It did help, working now.
          Also I would like to second Olivier's comment on your documentation, it is a model for others.

          Thanks,

          Tom

          Show
          brettonwoods Tom Burns added a comment - It did help, working now. Also I would like to second Olivier's comment on your documentation, it is a model for others. Thanks, Tom
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          ==== BETTER COMMENT: WE == COMMUNITY ====

          Since the community decided to move most specialpurpose components to Apache Extra (see slim down effort), the best place for this component is in Apache OFBiz Extra. Please read Apache Extra guidelines.

          Note the point 6:

          Each Apache Extras project is encouraged to publish their releases and complete the project tagging information available on Apache Extras.

          As explained in the guidelines, The Apache OFBiz Extra projects will not be official part of the Apache OFBiz project. The goals are pragmatic (at least as I see it):

          • Components which are not mandatory (not supposed to be used in each project powered by OFBiz) can be moved to Apache Extra.
          • With limited Apache OFBiz project human resources, it's a way to delegate the code and releases responsabilities to specific communities

          But this does not mean that Apache OFBiz Extra projects will be forgotten or neglected by the Apache OFBiz project. Now I feel something is missing in the scheme so far: how to relate an Apache OFBiz Extra project release to an Apache OFBiz release? I think it should part of the Apache OFBiz Extra project community to indicate which OFBiz release/s (trunk included) are compatible with the component. This is kind of part of the guidelines point 6 in my opininon. If this is correctly maintained it could be used by the Apache OFBiz project when releasing, at least we could have a link to the Apache OFBiz Extra projects page. Same from the Apache OFBiz project site main page, a reference should be added. I will do so soon, but will wait that at least a complete usable Apache OFBiz Extra project exists there. The Neogia team made a first effor but nothing was committed yet, the Solr component could be the first to engage the movement. With the ApacheConEu coming it could be also a good way to share about this point.

          Also I'd like to have a way to know the value of an Apache OFBiz Extra project, as experienced by its users. For the moment I see only one way to do this: recommend it on Google + (see in each component Project information). Of course this is a very limited way, but at least a mean.

          So it would be great if you Paul, Pascal and Jonatan (at least for now, the community will hopefully grow from comments above) could move the Solr component into the Apache OFBiz Extra projects. This would begin the dynamic, completing the pending work initiated by the Neogia team. Hopefully more will come later. For instance I read a lof about Big Fish, why not sharing in Apache OFBiz Extra projects also?

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited ==== BETTER COMMENT: WE == COMMUNITY ==== Since the community decided to move most specialpurpose components to Apache Extra ( see slim down effort ), the best place for this component is in Apache OFBiz Extra . Please read Apache Extra guidelines . Note the point 6: Each Apache Extras project is encouraged to publish their releases and complete the project tagging information available on Apache Extras. As explained in the guidelines, The Apache OFBiz Extra projects will not be official part of the Apache OFBiz project. The goals are pragmatic (at least as I see it): Components which are not mandatory (not supposed to be used in each project powered by OFBiz) can be moved to Apache Extra. With limited Apache OFBiz project human resources, it's a way to delegate the code and releases responsabilities to specific communities But this does not mean that Apache OFBiz Extra projects will be forgotten or neglected by the Apache OFBiz project. Now I feel something is missing in the scheme so far: how to relate an Apache OFBiz Extra project release to an Apache OFBiz release? I think it should part of the Apache OFBiz Extra project community to indicate which OFBiz release/s (trunk included) are compatible with the component. This is kind of part of the guidelines point 6 in my opininon. If this is correctly maintained it could be used by the Apache OFBiz project when releasing, at least we could have a link to the Apache OFBiz Extra projects page . Same from the Apache OFBiz project site main page , a reference should be added. I will do so soon, but will wait that at least a complete usable Apache OFBiz Extra project exists there. The Neogia team made a first effor but nothing was committed yet, the Solr component could be the first to engage the movement. With the ApacheConEu coming it could be also a good way to share about this point. Also I'd like to have a way to know the value of an Apache OFBiz Extra project, as experienced by its users. For the moment I see only one way to do this: recommend it on Google + (see in each component Project information). Of course this is a very limited way, but at least a mean. So it would be great if you Paul, Pascal and Jonatan (at least for now, the community will hopefully grow from comments above) could move the Solr component into the Apache OFBiz Extra projects. This would begin the dynamic, completing the pending work initiated by the Neogia team. Hopefully more will come later. For instance I read a lof about Big Fish, why not sharing in Apache OFBiz Extra projects also?
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Few notes after 1st quick review:

          • The Java classes use
            • packages "package com.ilscipio.solr". I think it's not a pb, since it wil not be official part-of/embedded-in OFBiz. To confirm packages names with PMC. In case of issue they could be something like org.apacheextras.ofbiz. or org.apache-extras.ofbiz. (for more read this thread). For me personally, it's ok. Like also <vendor>Ilscipio GmbH</vendor> in solrservices.xml (which would not be tolerated in OFBiz OOTB)
            • miss ASL2 headers (I guess you want to make it ASL2 licensed right?)
          • solrconfig.properties miss ASL2 header
          • <project name="OFBiz - Ecommerce Component" default="jar" basedir=".">
            should be rather
            <project name="OFBiz - Solr Component" default="jar" basedir=".">
          • jquery-1.4.3.min.js in solr\webapp\admin\ is part of Solr, right (all the webapp I guess?)? Else it should not be part of the component except if it supposed to work alone (solr standalone), jquery is already in OFBIz OOTB (different version by releases...)
          • solr\data\ dir could be removed (are we supposed to have OFBiz data proper to OFBiz?), right?
          • bin dir should be removed

          Well done guys

          Show
          jacques.le.roux Jacques Le Roux added a comment - Few notes after 1st quick review: The Java classes use packages "package com.ilscipio.solr". I think it's not a pb, since it wil not be official part-of/embedded-in OFBiz. To confirm packages names with PMC. In case of issue they could be something like org.apacheextras.ofbiz. or org.apache-extras.ofbiz. (for more read this thread ). For me personally, it's ok. Like also <vendor>Ilscipio GmbH</vendor> in solrservices.xml (which would not be tolerated in OFBiz OOTB) miss ASL2 headers (I guess you want to make it ASL2 licensed right?) solrconfig.properties miss ASL2 header <project name="OFBiz - Ecommerce Component" default="jar" basedir="."> should be rather <project name="OFBiz - Solr Component" default="jar" basedir="."> jquery-1.4.3.min.js in solr\webapp\admin\ is part of Solr, right (all the webapp I guess?)? Else it should not be part of the component except if it supposed to work alone (solr standalone), jquery is already in OFBIz OOTB (different version by releases...) solr\data\ dir could be removed (are we supposed to have OFBiz data proper to OFBiz?), right? bin dir should be removed Well done guys
          Hide
          madppiper Paul Piper added a comment -

          I missed your point on Apache Extra. With the discussion still ongoing I think we should wait before migrating this anywhere.

          Show
          madppiper Paul Piper added a comment - I missed your point on Apache Extra. With the discussion still ongoing I think we should wait before migrating this anywhere.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Sure there is no hurry, but we need to discuss it and get ahead I agree we have 1st to discuss about existing speicalpurpose components. Some are obvious to me, like moving ebay and google components to Extras, but maybe others will be against, etc. I hope though, we will quickly find a common ground. By consensus I mean, to avoid commits wars like we had in the past , and even votes if possible.

          Show
          jacques.le.roux Jacques Le Roux added a comment - Sure there is no hurry, but we need to discuss it and get ahead I agree we have 1st to discuss about existing speicalpurpose components. Some are obvious to me, like moving ebay and google components to Extras, but maybe others will be against, etc. I hope though, we will quickly find a common ground. By consensus I mean, to avoid commits wars like we had in the past , and even votes if possible.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Jacopo made a proposition and I believe it's what should be done.

          ========================
          BTW, some time ago I also proposed an alternative path: see email with subject "[PROPOSAL] from specialpurpose to extras": to that I can add that we could provide two set of ant scripts, one similar to the one we have that builds/tests everything (framework+applications+specialpurpose) and one (the default) that only builds/tests the framework+applications; the release branches may only contain the framework+applications and separate releases of specialpurpose applications could be voted/released at different time. This approach may reach two goals:
          1) slim down the "main" code that the community is more focused to improve/maintain/release
          2) keep under the OFBiz community the ownership of all the other specialpurpose components; if one of them will get more attention and interest and could grow in quality or it is generic enough we could decide to move it to the release branch (maybe move it to applications)
          ========================
          Jacopo

          The Solr component could then follow that way in specialpurpose, since it fits well with eCommerce.

          Show
          jacques.le.roux Jacques Le Roux added a comment - Jacopo made a proposition and I believe it's what should be done. ======================== BTW, some time ago I also proposed an alternative path: see email with subject " [PROPOSAL] from specialpurpose to extras": to that I can add that we could provide two set of ant scripts, one similar to the one we have that builds/tests everything (framework+applications+specialpurpose) and one (the default) that only builds/tests the framework+applications; the release branches may only contain the framework+applications and separate releases of specialpurpose applications could be voted/released at different time. This approach may reach two goals: 1) slim down the "main" code that the community is more focused to improve/maintain/release 2) keep under the OFBiz community the ownership of all the other specialpurpose components; if one of them will get more attention and interest and could grow in quality or it is generic enough we could decide to move it to the release branch (maybe move it to applications) ======================== Jacopo The Solr component could then follow that way in specialpurpose, since it fits well with eCommerce.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Hi guys,

          I have been working a bit on the solr component to have it working as a specialpurpose component instead of hotdeploy.
          At least the the solr webapp admin UI works, I did not test the other functionalities, but saw that in specialpurpose\solr\data folders index and spellchecker were created with files in.

          1. I simply added the solr home in start files (ant, bat sh). It's commented out by default. I commentted it out, because it will be straightforward when creating releases (which have only ecommerce component). But this is a bummer, because errors show when the solr component is loaded (no real impacts, but not real clean). If we decide to put solr with ecommerce (which makes sense IMO) then we would have solr loading cleanly as default. Another option though is to make as if solr is always loaded by default, even if it's not there. Putting -Dsolr.solr.home=specialpurpose/solr as JVM arg even it it's not used is not really a problem... I will discuss this on dev ML, I think the last argument is valid!
          2. I tried hard to find a way of using JNDI instead, but did not find one. I tried many ways, notably:
            • using <env-entry> in web.xml (which BTW would need to be put on bottom of file)
            • finding a Solr ContextFactory, something like SolrInitialContextFactory, nope (see SolrResourceLoader.locateSolrHome()). We could maybe create something specific in OFBiz, but looks like a hammer to kill a fly to me.
            • creating a solr.xml in runtime\catalina\conf\Catalina\localhost (referencing a specialpurpose\solr\lib\apache-solr-solrj-3.4.0.jar instead of a war, does not make sense anyway since it's to load it )
          3. Is there a reason to use Solr 3.4.0 instead of a newer version?
          4. BTW I wondered about this, what is supposed to be uiloginservice.build.xml ?
            D:\workspace\ofbizClean\specialpurpose\solr>ant clean build
            Buildfile: D:\workspace\ofbizClean\specialpurpose\solr\build.xml
            
            clean:
               [delete] Deleting directory D:\workspace\ofbizClean\specialpurpose\solr\build
            
            check-uiloginservice:
            
            clean-uiloginservice:
            
            build:
            
            init:
            
            prepare:
                [mkdir] Created dir: D:\workspace\ofbizClean\specialpurpose\solr\build\classes
                [mkdir] Created dir: D:\workspace\ofbizClean\specialpurpose\solr\build\lib
            
            classes:
              [javac16] Compiling 5 source files to D:\workspace\ofbizClean\specialpurpose\solr\build\classes
            
            jar:
                  [jar] Building jar: D:\workspace\ofbizClean\specialpurpose\solr\build\lib\ofbiz-solr.jar
            
            build-uiloginservice:
            
            BUILD FAILED
            D:\workspace\ofbizClean\specialpurpose\solr\build.xml:68: The following error occurred while executing this line:
            D:\workspace\ofbizClean\specialpurpose\solr\build.xml:61: Invalid file: D:\workspace\ofbizClean\specialpurpose\solr\uiloginservice.build.xml
            
            Total time: 2 seconds
            
            D:\workspace\ofbizClean\specialpurpose\solr>ant clean
            Buildfile: D:\workspace\ofbizClean\specialpurpose\solr\build.xml
            
            clean:
               [delete] Deleting directory D:\workspace\ofbizClean\specialpurpose\solr\build
            
            check-uiloginservice:
            
            clean-uiloginservice:
            
            BUILD SUCCESSFUL
            Total time: 0 seconds
            
            D:\workspace\ofbizClean\specialpurpose\solr>ant
            Buildfile: D:\workspace\ofbizClean\specialpurpose\solr\build.xml
            
            init:
            
            prepare:
                [mkdir] Created dir: D:\workspace\ofbizClean\specialpurpose\solr\build\classes
                [mkdir] Created dir: D:\workspace\ofbizClean\specialpurpose\solr\build\lib
            
            classes:
              [javac16] Compiling 5 source files to D:\workspace\ofbizClean\specialpurpose\solr\build\classes
            
            jar:
                  [jar] Building jar: D:\workspace\ofbizClean\specialpurpose\solr\build\lib\ofbiz-solr.jar
            
            BUILD SUCCESSFUL
            Total time: 1 second
            
          5. I removed slf4j-api-1.6.1.jar and slf4j-jdk14-1.6.1.jar from specialpurpose\solr\webapp\WEB-INF\lib because of
            SLF4J: Class path contains multiple SLF4J bindings.
            SLF4J: Found binding in [jar:file:/D:/workspace/ofbizClean/framework/base/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
            SLF4J: Found binding in [jar:file:/D:/workspace/ofbizClean/framework/base/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
            SLF4J: Found binding in [jar:file:/D:/workspace/ofbizClean/specialpurpose/solr/webapp/WEB-INF/lib/slf4j-jdk14-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
            SLF4J: Found binding in [jar:file:/D:/workspace/ofbizClean/specialpurpose/solr/webapp/WEB-INF/lib/slf4j-jdk14-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
            SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
          6. There is this warning, but it can safely be ignored
            Warning : Module classpath component 'D:\workspace\ofbizClean\specialpurpose\solr\src*' is not valid and will be ignored...

          Please find attached "solr in specialpurpose.zip" (with several changes for specialpurpose adaptation, notably the README.txt file) and "OFBIZ-5042 - solr in specialpurpose.patch". Please test and confirm all is also well with you, while I discuss the sorl home point on dev ML, thanks!

          Note: I like the Assistance section idea at bottom of Solr admin page!

          Show
          jacques.le.roux Jacques Le Roux added a comment - Hi guys, I have been working a bit on the solr component to have it working as a specialpurpose component instead of hotdeploy. At least the the solr webapp admin UI works, I did not test the other functionalities, but saw that in specialpurpose\solr\data folders index and spellchecker were created with files in. I simply added the solr home in start files (ant, bat sh). It's commented out by default. I commentted it out, because it will be straightforward when creating releases (which have only ecommerce component). But this is a bummer, because errors show when the solr component is loaded (no real impacts, but not real clean). If we decide to put solr with ecommerce (which makes sense IMO) then we would have solr loading cleanly as default. Another option though is to make as if solr is always loaded by default, even if it's not there. Putting -Dsolr.solr.home=specialpurpose/solr as JVM arg even it it's not used is not really a problem... I will discuss this on dev ML, I think the last argument is valid! I tried hard to find a way of using JNDI instead, but did not find one. I tried many ways, notably: using <env-entry> in web.xml (which BTW would need to be put on bottom of file) finding a Solr ContextFactory, something like SolrInitialContextFactory, nope (see SolrResourceLoader.locateSolrHome()) . We could maybe create something specific in OFBiz, but looks like a hammer to kill a fly to me. creating a solr.xml in runtime\catalina\conf\Catalina\localhost (referencing a specialpurpose\solr\lib\apache-solr-solrj-3.4.0.jar instead of a war, does not make sense anyway since it's to load it ) Is there a reason to use Solr 3.4.0 instead of a newer version? BTW I wondered about this, what is supposed to be uiloginservice.build.xml ? D:\workspace\ofbizClean\specialpurpose\solr>ant clean build Buildfile: D:\workspace\ofbizClean\specialpurpose\solr\build.xml clean: [delete] Deleting directory D:\workspace\ofbizClean\specialpurpose\solr\build check-uiloginservice: clean-uiloginservice: build: init: prepare: [mkdir] Created dir: D:\workspace\ofbizClean\specialpurpose\solr\build\classes [mkdir] Created dir: D:\workspace\ofbizClean\specialpurpose\solr\build\lib classes: [javac16] Compiling 5 source files to D:\workspace\ofbizClean\specialpurpose\solr\build\classes jar: [jar] Building jar: D:\workspace\ofbizClean\specialpurpose\solr\build\lib\ofbiz-solr.jar build-uiloginservice: BUILD FAILED D:\workspace\ofbizClean\specialpurpose\solr\build.xml:68: The following error occurred while executing this line: D:\workspace\ofbizClean\specialpurpose\solr\build.xml:61: Invalid file: D:\workspace\ofbizClean\specialpurpose\solr\uiloginservice.build.xml Total time: 2 seconds D:\workspace\ofbizClean\specialpurpose\solr>ant clean Buildfile: D:\workspace\ofbizClean\specialpurpose\solr\build.xml clean: [delete] Deleting directory D:\workspace\ofbizClean\specialpurpose\solr\build check-uiloginservice: clean-uiloginservice: BUILD SUCCESSFUL Total time: 0 seconds D:\workspace\ofbizClean\specialpurpose\solr>ant Buildfile: D:\workspace\ofbizClean\specialpurpose\solr\build.xml init: prepare: [mkdir] Created dir: D:\workspace\ofbizClean\specialpurpose\solr\build\classes [mkdir] Created dir: D:\workspace\ofbizClean\specialpurpose\solr\build\lib classes: [javac16] Compiling 5 source files to D:\workspace\ofbizClean\specialpurpose\solr\build\classes jar: [jar] Building jar: D:\workspace\ofbizClean\specialpurpose\solr\build\lib\ofbiz-solr.jar BUILD SUCCESSFUL Total time: 1 second I removed slf4j-api-1.6.1.jar and slf4j-jdk14-1.6.1.jar from specialpurpose\solr\webapp\WEB-INF\lib because of SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/D:/workspace/ofbizClean/framework/base/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/D:/workspace/ofbizClean/framework/base/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/D:/workspace/ofbizClean/specialpurpose/solr/webapp/WEB-INF/lib/slf4j-jdk14-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/D:/workspace/ofbizClean/specialpurpose/solr/webapp/WEB-INF/lib/slf4j-jdk14-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. There is this warning, but it can safely be ignored Warning : Module classpath component 'D:\workspace\ofbizClean\specialpurpose\solr\src*' is not valid and will be ignored... Please find attached "solr in specialpurpose.zip" (with several changes for specialpurpose adaptation, notably the README.txt file) and " OFBIZ-5042 - solr in specialpurpose.patch". Please test and confirm all is also well with you, while I discuss the sorl home point on dev ML, thanks! Note: I like the Assistance section idea at bottom of Solr admin page!
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          About the last note above "I like the Assistance section idea at bottom of Solr admin page!"

          What I mean is the links they provide are useful, we could provide the same on bottom of pages in ERP side of OFBiz

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited About the last note above "I like the Assistance section idea at bottom of Solr admin page!" What I mean is the links they provide are useful, we could provide the same on bottom of pages in ERP side of OFBiz
          Hide
          madppiper Paul Piper added a comment -

          Hi Jacques,

          looks good. Though we may want to upgrade to the latest solr version (4.4). Ours was 3.4 when we did this and alot of time has passed since. They updated the gui heavily (amongst other things) and so an upgrade would make sense.

          Show
          madppiper Paul Piper added a comment - Hi Jacques, looks good. Though we may want to upgrade to the latest solr version (4.4). Ours was 3.4 when we did this and alot of time has passed since. They updated the gui heavily (amongst other things) and so an upgrade would make sense.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Indeed, interesting https://lucene.apache.org/solr/screenshots.html. I will try to upgrade, thanks!

          Show
          jacques.le.roux Jacques Le Roux added a comment - Indeed, interesting https://lucene.apache.org/solr/screenshots.html . I will try to upgrade, thanks!
          Hide
          pfm.smits Pierre Smits added a comment -

          Given that the admin panel is intended as a backend configuration/check/monitoring application it would be wise to not use port 8080 as the default, but - like any other backend component - rather the 8443 port

          Show
          pfm.smits Pierre Smits added a comment - Given that the admin panel is intended as a backend configuration/check/monitoring application it would be wise to not use port 8080 as the default, but - like any other backend component - rather the 8443 port
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          This seems doable and a good idea Pierre, I will try, thanks! BTW another port instance to take into account for OFBIZ-4794 which I have almost ready for trunk but pending for releases...

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited This seems doable and a good idea Pierre, I will try, thanks! BTW another port instance to take into account for OFBIZ-4794 which I have almost ready for trunk but pending for releases...
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Hi Guys,

          Before I dive deeper in code, I noticed that in solr CategoryUtil class getCategoryTrail() is almost the same than one version which was contributed before in Product CategoryWorker class. Is it an improved version, or only related?

          Show
          jacques.le.roux Jacques Le Roux added a comment - Hi Guys, Before I dive deeper in code, I noticed that in solr CategoryUtil class getCategoryTrail() is almost the same than one version which was contributed before in Product CategoryWorker class. Is it an improved version, or only related?
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I upgraded to solr 4.4.0. The admin view works w/o issues it seems. I only found warnings like this in log

          2013-08-03 14:13:11,983 (coreLoadExecutor-3-thread-1) [FieldTypePluginLoader.java:411:WARN ] TokenizerFactory is using deprecated LUCENE_34 emulation. You should at some point declare and reindex to at least 4.0, because 3.x emulation is deprecated and will be removed in 5.0.

          Even after removing the data folder where index are. Anyway not a big deal for now it seems.

          Note: I also changed the deprecated CommonsHttpSolrServer to recommended HttpSolrServer. I saw also that SolrQuery.setSortField() was deprecated and easily replaced by SolrQuery.setSort(). I did not find any other deprecation in solr component code nor any use of TokenizerFactory in whole OFBiz code. So I guess above warning are related to something loaded with catalina. We can neglect for now, when it will be obvious wrong we will change it...

          Show
          jacques.le.roux Jacques Le Roux added a comment - I upgraded to solr 4.4.0. The admin view works w/o issues it seems. I only found warnings like this in log 2013-08-03 14:13:11,983 (coreLoadExecutor-3-thread-1) [FieldTypePluginLoader.java:411:WARN ] TokenizerFactory is using deprecated LUCENE_34 emulation. You should at some point declare and reindex to at least 4.0, because 3.x emulation is deprecated and will be removed in 5.0. Even after removing the data folder where index are. Anyway not a big deal for now it seems. Note: I also changed the deprecated CommonsHttpSolrServer to recommended HttpSolrServer. I saw also that SolrQuery.setSortField() was deprecated and easily replaced by SolrQuery.setSort() . I did not find any other deprecation in solr component code nor any use of TokenizerFactory in whole OFBiz code. So I guess above warning are related to something loaded with catalina. We can neglect for now, when it will be obvious wrong we will change it...
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          About changing security access to solr admin. It quite needed because you can change things from there. But it turned that solr admin webapp as is is only a demo. So we will need to follow https://wiki.apache.org/solr/SolrSecurity#Path_Based_Authentication. It's worth noting deeper issues will only be fixed in 4.5+ versions https://issues.apache.org/jira/browse/SOLR-4470 but it seems we can apply the last version of SOLR-4470_branch_4x_r1454444.patch .

          So still some work there to be committed in OFBiz...

          Show
          jacques.le.roux Jacques Le Roux added a comment - About changing security access to solr admin. It quite needed because you can change things from there. But it turned that solr admin webapp as is is only a demo. So we will need to follow https://wiki.apache.org/solr/SolrSecurity#Path_Based_Authentication . It's worth noting deeper issues will only be fixed in 4.5+ versions https://issues.apache.org/jira/browse/SOLR-4470 but it seems we can apply the last version of SOLR-4470 _branch_4x_r1454444.patch . So still some work there to be committed in OFBiz...
          Hide
          pfm.smits Pierre Smits added a comment -

          Jacques,

          Is this component using the lucene*.jar files that are currently in the content component of OFBiz? The lucene libraries are also at release lvl 4.0, if I am correct.

          If not, which I suspect, why not bring this to the lower level and have the indexing component integrated in framework (and have one set of libraries to maintain) so that this kind of (search) functionality can be reused in all components at higher levels (applications, specialpurpose, hot-deploy)?

          What are your thoughts on this?

          Regards,

          Pierre

          Show
          pfm.smits Pierre Smits added a comment - Jacques, Is this component using the lucene*.jar files that are currently in the content component of OFBiz? The lucene libraries are also at release lvl 4.0, if I am correct. If not, which I suspect, why not bring this to the lower level and have the indexing component integrated in framework (and have one set of libraries to maintain) so that this kind of (search) functionality can be reused in all components at higher levels (applications, specialpurpose, hot-deploy)? What are your thoughts on this? Regards, Pierre
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Pierre,

          I have not considered it, I guess those who contributed (Paul, Pascal and Jonatan from ilscipio) have a better idea on that. I will wait their feedback...

          Show
          jacques.le.roux Jacques Le Roux added a comment - Pierre, I have not considered it, I guess those who contributed (Paul, Pascal and Jonatan from ilscipio) have a better idea on that. I will wait their feedback...
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          While looking at integrating log4j2 in OFBiz whith Solr 4.4 already in place, I though about the possible (minor) issue (already crossed) with SLF4J. But there are any SLF4J lib Solr 4.4. Also (unrelated) good news there is a SLF4J Bridge for log4j2

          This remembered me your question Pierre. I believe there are 2 parts in the solr component: the Solr admin wich stays at specialpurpose\solr\webapp and the indexing part which relies on solr-solrj-4.4.0.jar in specialpurpose\solr\lib. So it could be possible to move this last lib from solr to framework\base\lib. But this would be a very small part of the problem of replacing the current internal Lucene indexing by Solr. If I have well understood the current contribution addresses only the case of ecommerce products indexing. There are maybe good reasons (apart the work to be done) for that, I again let the contributors confirm...

          Show
          jacques.le.roux Jacques Le Roux added a comment - While looking at integrating log4j2 in OFBiz whith Solr 4.4 already in place, I though about the possible (minor) issue (already crossed) with SLF4J. But there are any SLF4J lib Solr 4.4. Also (unrelated) good news there is a SLF4J Bridge for log4j2 This remembered me your question Pierre. I believe there are 2 parts in the solr component: the Solr admin wich stays at specialpurpose\solr\webapp and the indexing part which relies on solr-solrj-4.4.0.jar in specialpurpose\solr\lib. So it could be possible to move this last lib from solr to framework\base\lib. But this would be a very small part of the problem of replacing the current internal Lucene indexing by Solr. If I have well understood the current contribution addresses only the case of ecommerce products indexing. There are maybe good reasons (apart the work to be done) for that, I again let the contributors confirm...
          Hide
          madppiper Paul Piper added a comment -

          I think the idea would be lovely, Pierre. The engine can be used to maintain an index for high-performance lookups. In theory we could use it for indexing much more than just product data as well. That way we would rely on the full db queries only for whenever we require some very detailed information - it would certainly be an instant performance boost for all apps. We would have to redo the solr schema.xml, however and modify it with more data in mind.

          Show
          madppiper Paul Piper added a comment - I think the idea would be lovely, Pierre. The engine can be used to maintain an index for high-performance lookups. In theory we could use it for indexing much more than just product data as well. That way we would rely on the full db queries only for whenever we require some very detailed information - it would certainly be an instant performance boost for all apps. We would have to redo the solr schema.xml, however and modify it with more data in mind.
          Hide
          pfm.smits Pierre Smits added a comment -

          Not only a performance boost for all apps, Paul.

          But also a functionality boost. Think of all content stored in the content app related to products (like flyers, manuals, etc), but also resumes related to HR, notes to workeffort and more. Or even plain webcontent.

          Yes, I agree that the solr schema.xml must be modified for that. But the greater part is generic. We would only have to figure out how to avoid contamination of query results.

          Show
          pfm.smits Pierre Smits added a comment - Not only a performance boost for all apps, Paul. But also a functionality boost. Think of all content stored in the content app related to products (like flyers, manuals, etc), but also resumes related to HR, notes to workeffort and more. Or even plain webcontent. Yes, I agree that the solr schema.xml must be modified for that. But the greater part is generic. We would only have to figure out how to avoid contamination of query results.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          After giving up on using JNI (instead of a JVM arg) for Solr home, I wanted to do more work to integrate Solr change for next coming versions. I mean 5.x which are here documented but I finally postponed them because it's not blocking yet, and I have other priorities. Moreover, if we want to integrate Solr further maybe no needs to change that now. Note that this also concerns changes from 3.x to 4.x in solrconfig.xml. As I said above, I did all the others minor but mandatory changes. Note that some changes will become mandatory coming with 5.x versions and will replace those settings. Here is a how-to. I have also noticed this stackoverflow thread but did not use it.

          BTW I found this easy to read (but incomplete) change.log. Else you always can refer to the CHANGES.txt in your package.

          So here are attached a new zipped component (splitted with Winzip due to 10 MB max upload in Jira, you need Winzip to "open solr in specialpurpose splitted.zip" and the 2 others realted files as a sole file) and the related patch for Solr 4.4 version.

          Show
          jacques.le.roux Jacques Le Roux added a comment - After giving up on using JNI (instead of a JVM arg) for Solr home, I wanted to do more work to integrate Solr change for next coming versions. I mean 5.x which are here documented but I finally postponed them because it's not blocking yet, and I have other priorities. Moreover, if we want to integrate Solr further maybe no needs to change that now. Note that this also concerns changes from 3.x to 4.x in solrconfig.xml . As I said above, I did all the others minor but mandatory changes . Note that some changes will become mandatory coming with 5.x versions and will replace those settings . Here is a how-to . I have also noticed this stackoverflow thread but did not use it . BTW I found this easy to read (but incomplete) change.log . Else you always can refer to the CHANGES.txt in your package. So here are attached a new zipped component (splitted with Winzip due to 10 MB max upload in Jira, you need Winzip to "open solr in specialpurpose splitted.zip" and the 2 others realted files as a sole file) and the related patch for Solr 4.4 version.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          About Pierre's concern about http access to Solr admin UI: actually it's not the most serious problem. You can also access the Solr admin UI also using https://localhost:8443/solr/admin/. I wanted to add a link from Webtools.

          But then I thought about possible security issues, because OOTB Solr does not provide any authenticated access. There are possible solutions explained in the Solr wiki. I began to look at using the <security-constraint> way, but maybe it would be better to wait for the 4.5 version because of SOLR-4470, though there is also a patch there for 4.4.

          I also noticed there are properties solr.webapp.portOverride and solr.webapp.path in solrconfig.properties but those are only used by the internal solr component (ie not the Solr webapp admin).

          Show
          jacques.le.roux Jacques Le Roux added a comment - About Pierre's concern about http access to Solr admin UI: actually it's not the most serious problem. You can also access the Solr admin UI also using https://localhost:8443/solr/admin/ . I wanted to add a link from Webtools. But then I thought about possible security issues, because OOTB Solr does not provide any authenticated access. There are possible solutions explained in the Solr wiki . I began to look at using the <security-constraint> way, but maybe it would be better to wait for the 4.5 version because of SOLR-4470 , though there is also a patch there for 4.4. I also noticed there are properties solr.webapp.portOverride and solr.webapp.path in solrconfig.properties but those are only used by the internal solr component (ie not the Solr webapp admin).
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I forgot to put small changes in the specialpurpose/LICENSE file, this new patch contains them

          Show
          jacques.le.roux Jacques Le Roux added a comment - I forgot to put small changes in the specialpurpose/LICENSE file, this new patch contains them
          Hide
          djjava Tri Duc Vo added a comment -

          Thank All
          Cheers

          Show
          djjava Tri Duc Vo added a comment - Thank All Cheers
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          I put as resolved/incomplete to facilitate searches with "My resolved pending" own filter

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited I put as resolved/incomplete to facilitate searches with "My resolved pending" own filter
          Hide
          jacopoc Jacopo Cappellato added a comment -

          In order to simplify efforts like the one in this Jira tickets and to simplify the management of conflicts caused by different versions of Solr/Lucene jars, I am working at extracting from the OFBiz content component the dependencies on Lucene: I am creating a special purpose/lucene component to host them.
          In this way it will be easier to compare the ootb Lucene integration with new ones (based on Lucene or Solr) and to switch between them.

          Show
          jacopoc Jacopo Cappellato added a comment - In order to simplify efforts like the one in this Jira tickets and to simplify the management of conflicts caused by different versions of Solr/Lucene jars, I am working at extracting from the OFBiz content component the dependencies on Lucene: I am creating a special purpose/lucene component to host them. In this way it will be easier to compare the ootb Lucene integration with new ones (based on Lucene or Solr) and to switch between them.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Good news, thanks for the effort Jacopo!

          Show
          jacques.le.roux Jacques Le Roux added a comment - Good news, thanks for the effort Jacopo!
          Hide
          sumitp Sumit Pandit added a comment -

          Hi Jacques, trying to extract files : "solr in specialpurpose splitted.z01" & "solr in specialpurpose splitted.z02", but couldn't. Can you please upload a zip version of same, that would be a great help!

          Show
          sumitp Sumit Pandit added a comment - Hi Jacques, trying to extract files : "solr in specialpurpose splitted.z01" & "solr in specialpurpose splitted.z02", but couldn't. Can you please upload a zip version of same, that would be a great help!
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          Hi Sumit,

          Did you try to download the 3 zipped files (solr in specialpurpose splitted.zip, solr in specialpurpose splitted.z01, solr in specialpurpose splitted.z02) in the same place and open specialpurpose splitted.zip with Winzip?

          If it does not work, I have put a copy of the plain zip file at http://www.les7arts.com/solr_in_specialpurpose.zip

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited Hi Sumit, Did you try to download the 3 zipped files (solr in specialpurpose splitted.zip, solr in specialpurpose splitted.z01, solr in specialpurpose splitted.z02) in the same place and open specialpurpose splitted.zip with Winzip? If it does not work, I have put a copy of the plain zip file at http://www.les7arts.com/solr_in_specialpurpose.zip
          Hide
          sumitp Sumit Pandit added a comment - - edited

          Hi Jacques,

          Thanks for sharing the link. It helps!
          BTW got success in extract the attachments in following way -

          a. Downloaded three files;
          b. Executed :

          zip -s- solr\ in\ specialpurpose\ splitted.zip -O solr_in_specialpurpose_splitted.zip
          

          c. Command has generated combined zip files as solr_in_specialpurpose_splitted.zip
          d. Then, unzip command gave an expected output -

          unzip solr_in_specialpurpose_splitted.zip
          
          Show
          sumitp Sumit Pandit added a comment - - edited Hi Jacques, Thanks for sharing the link. It helps! BTW got success in extract the attachments in following way - a. Downloaded three files; b. Executed : zip -s- solr\ in\ specialpurpose\ splitted.zip -O solr_in_specialpurpose_splitted.zip c. Command has generated combined zip files as solr_in_specialpurpose_splitted.zip d. Then, unzip command gave an expected output - unzip solr_in_specialpurpose_splitted.zip
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Thanks Sumit,

          Yes, forgot there is no Winzip for *nix

          I guess the command is rather?

           zip -s solr\ in\ specialpurpose\ splitted.zip -O solr_in_specialpurpose_splitted.zip
           

          (you surrounded s with -, which leads to strike).

          If not, surrounding the line with

          { code }

          or

          { noformat }

          will do it

          Show
          jacques.le.roux Jacques Le Roux added a comment - Thanks Sumit, Yes, forgot there is no Winzip for *nix I guess the command is rather? zip -s solr\ in\ specialpurpose\ splitted.zip -O solr_in_specialpurpose_splitted.zip (you surrounded s with -, which leads to strike). If not, surrounding the line with { code } or { noformat } will do it
          Hide
          sumitp Sumit Pandit added a comment -

          Thanks Jacques for suggestion. modifying my comment!

          Show
          sumitp Sumit Pandit added a comment - Thanks Jacques for suggestion. modifying my comment!
          Show
          jacques.le.roux Jacques Le Roux added a comment - For anyone interested by the security reasons which made me stop before
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Some recent security fixes in Solr:

          Show
          jacques.le.roux Jacques Le Roux added a comment - Some recent security fixes in Solr: major SOLR-3895 medium SOLR-4881 SOLR-4882
          Hide
          pfm.smits Pierre Smits added a comment -

          What is still required to have this committed to trunk?
          This would be a great addition to the feature set of OFBiz, helping the adoption even more.

          Show
          pfm.smits Pierre Smits added a comment - What is still required to have this committed to trunk? This would be a great addition to the feature set of OFBiz, helping the adoption even more.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          It works as is, but I stopped because I thought we needed to verify the access to the admin module was secured
          Now, I believe I went to far and it could be used as is. As long as there is only 1 Solr node, which is what this patch provides

          More can be found in the SOLR-* links I put above...

          Show
          jacques.le.roux Jacques Le Roux added a comment - It works as is, but I stopped because I thought we needed to verify the access to the admin module was secured Now, I believe I went to far and it could be used as is. As long as there is only 1 Solr node, which is what this patch provides More can be found in the SOLR-* links I put above...
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          In other words, tests and feedbacks are welcome, notably on the security aspect, for the rest I'm quite confident.

          Show
          jacques.le.roux Jacques Le Roux added a comment - In other words, tests and feedbacks are welcome, notably on the security aspect, for the rest I'm quite confident.
          Hide
          sumitp Sumit Pandit added a comment -

          Hi Paul, Jacques,

          I have implemented solr for one of my project using your patch with customization in some of the services.
          I am facing a big problem, see if you could help me out in this, also please let me know if you would required additional input-

          • Feature implemented - Auto completer for keyword search.
          • Problem facing - It is wroking fine on local (with less then 100 test products). But while deployed on production (with 50000 live products). index documents are generated for for all products but suggester document is not generating for any of them.
          • Proving some part of logs below, See if you could suggest anything.
          • Configurations -
            1) SolrConfig.xml -
            <searchComponent name="suggest" class="solr.SpellCheckComponent">
            <str name="queryAnalyzerFieldType">auto_complete</str>
            <lst name="spellchecker">
            <str name="name">suggest</str>
            <str name="classname">org.apache.solr.spelling.suggest.Suggester</str>
            <str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookup</str>
            <str name="field">internalName</str>
            <float name="threshold">0.005</float>
            <str name="buildOnCommit">true</str>
            </lst>
            </searchComponent>
            <requestHandler class="solr.SearchHandler" name="/suggest">
            <lst name="defaults">
            <str name="spellcheck">true</str>
            <str name="spellcheck.dictionary">suggest</str>
            <str name="spellcheck.onlyMorePopular">false</str>
            <str name="spellcheck.count">5</str>
            <str name="spellcheck.collate">false</str>
            </lst>
            <arr name="components">
            <str>suggest</str>
            </arr>
            </requestHandler>
            ---------------------------------------------------
            2) Schema.xml
            <fieldType name="auto_complete" class="solr.TextField" positionIncrementGap="100">
            <analyzer type="index">
            <tokenizer class="solr.KeywordTokenizerFactory" />
            <filter class="solr.LowerCaseFilterFactory" />
            </analyzer>
            <analyzer type="query">
            <tokenizer class="solr.KeywordTokenizerFactory" />
            <filter class="solr.LowerCaseFilterFactory" />
            </analyzer>
            </fieldType>
            <field name="internalName" type="auto_complete" indexed="true" stored="true" multiValued="true"/>
            <copyField source="internalName" dest="text"/>
          • Logs - Observing a lot of entires of logs as below -
            2014-11-22 12:15:06,260 (http-bio-0.0.0.0-8080-exec-71) [ SolrDeletionPolicy.java:165:INFO ] newest commit generation = 1082
            2014-11-22 12:15:06,260 (http-bio-0.0.0.0-8080-exec-71) [ SolrCore.java:1622:WARN ] [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
            2014-11-22 12:15:06,261 (http-bio-0.0.0.0-8080-exec-71) [ SolrIndexSearcher.java:197:INFO ] Opening Searcher@15b7db9c main
            2014-11-22 12:15:06,261 (http-bio-0.0.0.0-8080-exec-71) [DirectUpdateHandler2.java:602:INFO ] end_commit_flush
            2014-11-22 12:15:06,262 (http-bio-0.0.0.0-8080-exec-112) [DirectUpdateHandler2.java:511:INFO ] start commit {,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
            2014-11-22 12:15:06,262 (http-bio-0.0.0.0-8080-exec-112) [DirectUpdateHandler2.java:551:INFO ] No uncommitted changes. Skipping IW.commit.
            2014-11-22 12:15:06,262 (http-bio-0.0.0.0-8080-exec-112) [ SolrCore.java:1622:WARN ] [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=3
            2014-11-22 12:15:06,263 (http-bio-0.0.0.0-8080-exec-112) [ SolrIndexSearcher.java:197:INFO ] Opening Searcher@711a6ab2 main
            2014-11-22 12:15:06,263 (http-bio-0.0.0.0-8080-exec-112) [DirectUpdateHandler2.java:602:INFO ] end_commit_flush
            2014-11-22 12:15:06,263 (http-bio-0.0.0.0-8080-exec-86) [DirectUpdateHandler2.java:511:INFO ] start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}

            2014-11-22 12:15:06,263 (http-bio-0.0.0.0-8080-exec-86) [DirectUpdateHandler2.java:551:INFO ] No uncommitted changes. Skipping IW.commit.
            2014-11-22 12:15:06,263 (http-bio-0.0.0.0-8080-exec-86) [ SolrCore.java:1622:WARN ] [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=4
            2014-11-22 12:15:06,264 (http-bio-0.0.0.0-8080-exec-86) [ SolrIndexSearcher.java:197:INFO ] Opening Searcher@55b75f27 main
            2014-11-22 12:15:06,264 (http-bio-0.0.0.0-8080-exec-86) [DirectUpdateHandler2.java:602:INFO ] end_commit_flush
            2014-11-22 12:15:06,264 (http-bio-0.0.0.0-8080-exec-140) [DirectUpdateHandler2.java:511:INFO ] start commit

            {,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}

            2014-11-22 12:15:06,265 (http-bio-0.0.0.0-8080-exec-140) [DirectUpdateHandler2.java:551:INFO ] No uncommitted changes. Skipping IW.commit.
            2014-11-22 12:15:06,265 (http-bio-0.0.0.0-8080-exec-140) [ SolrCore.java:1622:WARN ] [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=5

          Thanks in advance
          Sumit Pandit

          Show
          sumitp Sumit Pandit added a comment - Hi Paul, Jacques, I have implemented solr for one of my project using your patch with customization in some of the services. I am facing a big problem, see if you could help me out in this, also please let me know if you would required additional input- Feature implemented - Auto completer for keyword search. Problem facing - It is wroking fine on local (with less then 100 test products). But while deployed on production (with 50000 live products). index documents are generated for for all products but suggester document is not generating for any of them. Proving some part of logs below, See if you could suggest anything. Configurations - 1) SolrConfig.xml - <searchComponent name="suggest" class="solr.SpellCheckComponent"> <str name="queryAnalyzerFieldType">auto_complete</str> <lst name="spellchecker"> <str name="name">suggest</str> <str name="classname">org.apache.solr.spelling.suggest.Suggester</str> <str name="lookupImpl">org.apache.solr.spelling.suggest.tst.TSTLookup</str> <str name="field">internalName</str> <float name="threshold">0.005</float> <str name="buildOnCommit">true</str> </lst> </searchComponent> <requestHandler class="solr.SearchHandler" name="/suggest"> <lst name="defaults"> <str name="spellcheck">true</str> <str name="spellcheck.dictionary">suggest</str> <str name="spellcheck.onlyMorePopular">false</str> <str name="spellcheck.count">5</str> <str name="spellcheck.collate">false</str> </lst> <arr name="components"> <str>suggest</str> </arr> </requestHandler> --------------------------------------------------- 2) Schema.xml <fieldType name="auto_complete" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="solr.KeywordTokenizerFactory" /> <filter class="solr.LowerCaseFilterFactory" /> </analyzer> <analyzer type="query"> <tokenizer class="solr.KeywordTokenizerFactory" /> <filter class="solr.LowerCaseFilterFactory" /> </analyzer> </fieldType> <field name="internalName" type="auto_complete" indexed="true" stored="true" multiValued="true"/> <copyField source="internalName" dest="text"/> Logs - Observing a lot of entires of logs as below - 2014-11-22 12:15:06,260 (http-bio-0.0.0.0-8080-exec-71) [ SolrDeletionPolicy.java:165:INFO ] newest commit generation = 1082 2014-11-22 12:15:06,260 (http-bio-0.0.0.0-8080-exec-71) [ SolrCore.java:1622:WARN ] [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=2 2014-11-22 12:15:06,261 (http-bio-0.0.0.0-8080-exec-71) [ SolrIndexSearcher.java:197:INFO ] Opening Searcher@15b7db9c main 2014-11-22 12:15:06,261 (http-bio-0.0.0.0-8080-exec-71) [DirectUpdateHandler2.java:602:INFO ] end_commit_flush 2014-11-22 12:15:06,262 (http-bio-0.0.0.0-8080-exec-112) [DirectUpdateHandler2.java:511:INFO ] start commit {,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} 2014-11-22 12:15:06,262 (http-bio-0.0.0.0-8080-exec-112) [DirectUpdateHandler2.java:551:INFO ] No uncommitted changes. Skipping IW.commit. 2014-11-22 12:15:06,262 (http-bio-0.0.0.0-8080-exec-112) [ SolrCore.java:1622:WARN ] [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=3 2014-11-22 12:15:06,263 (http-bio-0.0.0.0-8080-exec-112) [ SolrIndexSearcher.java:197:INFO ] Opening Searcher@711a6ab2 main 2014-11-22 12:15:06,263 (http-bio-0.0.0.0-8080-exec-112) [DirectUpdateHandler2.java:602:INFO ] end_commit_flush 2014-11-22 12:15:06,263 (http-bio-0.0.0.0-8080-exec-86) [DirectUpdateHandler2.java:511:INFO ] start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} 2014-11-22 12:15:06,263 (http-bio-0.0.0.0-8080-exec-86) [DirectUpdateHandler2.java:551:INFO ] No uncommitted changes. Skipping IW.commit. 2014-11-22 12:15:06,263 (http-bio-0.0.0.0-8080-exec-86) [ SolrCore.java:1622:WARN ] [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=4 2014-11-22 12:15:06,264 (http-bio-0.0.0.0-8080-exec-86) [ SolrIndexSearcher.java:197:INFO ] Opening Searcher@55b75f27 main 2014-11-22 12:15:06,264 (http-bio-0.0.0.0-8080-exec-86) [DirectUpdateHandler2.java:602:INFO ] end_commit_flush 2014-11-22 12:15:06,264 (http-bio-0.0.0.0-8080-exec-140) [DirectUpdateHandler2.java:511:INFO ] start commit {,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} 2014-11-22 12:15:06,265 (http-bio-0.0.0.0-8080-exec-140) [DirectUpdateHandler2.java:551:INFO ] No uncommitted changes. Skipping IW.commit. 2014-11-22 12:15:06,265 (http-bio-0.0.0.0-8080-exec-140) [ SolrCore.java:1622:WARN ] [collection1] PERFORMANCE WARNING: Overlapping onDeckSearchers=5 Thanks in advance Sumit Pandit
          Hide
          madppiper Paul Piper added a comment -

          Hi,

          we run an installation of 2.5 Mio products and have no such issue. My best guess from your logs is that you seem to bombard the solr server with alot of requests. Please check how you reindex the solr tree. You may be using a eeca that commits products individually once they are added to the db. If that is the case, the solr server may simply be overwhelmed by the mass of individual publications. On our end we implemented a way to collect and publish products to the tree in bulks, which is better suited for large-scale implementations.

          See also: https://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Overlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F

          Regards,
          Paul

          Show
          madppiper Paul Piper added a comment - Hi, we run an installation of 2.5 Mio products and have no such issue. My best guess from your logs is that you seem to bombard the solr server with alot of requests. Please check how you reindex the solr tree. You may be using a eeca that commits products individually once they are added to the db. If that is the case, the solr server may simply be overwhelmed by the mass of individual publications. On our end we implemented a way to collect and publish products to the tree in bulks, which is better suited for large-scale implementations. See also: https://wiki.apache.org/solr/FAQ#What_does_.22PERFORMANCE_WARNING:_Overlapping_onDeckSearchers.3DX.22_mean_in_my_logs.3F Regards, Paul
          Hide
          sumitp Sumit Pandit added a comment -

          Thanks Paul, I will try your suggestion.

          Show
          sumitp Sumit Pandit added a comment - Thanks Paul, I will try your suggestion.
          Show
          jacques.le.roux Jacques Le Roux added a comment - Jus as a note here https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-3628
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Cross-site scripting (XSS) vulnerability in the Admin UI Plugin / Stats page in Apache Solr 4.x before 4.10.3 allows remote attackers to inject arbitrary web script or HTML via the fieldvaluecache object.

          Show
          jacques.le.roux Jacques Le Roux added a comment - Cross-site scripting (XSS) vulnerability in the Admin UI Plugin / Stats page in Apache Solr 4.x before 4.10.3 allows remote attackers to inject arbitrary web script or HTML via the fieldvaluecache object.
          Hide
          pfm.smits Pierre Smits added a comment -

          I would think that warrants an upgrade.

          Show
          pfm.smits Pierre Smits added a comment - I would think that warrants an upgrade.
          Hide
          sanjeev Sanjeev Gupta added a comment - - edited

          Hi All,
          I'm testing this patch on trunk (a slightly outdated release 1559083 on a windows machine ) but getting the following error ..
          Just wondering if anyone has faced this issue or can help me fix the issue ..

          2015-04-06 18:51:05,241 (catalina-startup-0) [ SolrDispatchFilter.java:116:INFO ] SolrDispatchFilter.init()
          2015-04-06 18:51:05,318 (catalina-startup-0) [ SolrResourceLoader.java:683:INFO ] No /solr/home in JNDI
          2015-04-06 18:51:05,318 (catalina-startup-0) [ SolrResourceLoader.java:693:INFO ] using system property solr.solr.home: specialpurpose/solr
          2015-04-06 18:51:05,323 (catalina-startup-0) [ SolrResourceLoader.java:115:INFO ] new SolrResourceLoader for directory: 'specialpurpose/solr\'
          2015-04-06 18:51:05,327 (catalina-startup-0) [ SolrResourceLoader.java:207:INFO ] Adding 'file:/D:/dwaCommerce/trunk-1559083/specialpurpose/solr/lib/solr-solrj-4.4.0.jar' to classloader
          2015-04-06 18:51:05,426 (catalina-startup-0) [ SolrDispatchFilter.java:127:ERROR] Could not start Solr. Check solr/home property and the logs
          2015-04-06 18:51:05,507 (catalina-startup-0) [      SolrException.java:119:ERROR] null:java.lang.ExceptionInInitializerError
                  at org.apache.solr.core.SolrResourceLoader.reloadLuceneSPI(SolrResourceLoader.java:184)
                  at org.apache.solr.core.SolrResourceLoader.<init>(SolrResourceLoader.java:121)
                  at org.apache.solr.core.SolrResourceLoader.<init>(SolrResourceLoader.java:235)
                  at org.apache.solr.core.CoreContainer.<init>(CoreContainer.java:129)
                  at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:139)
                  at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:122)
                  at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:281)
                  at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:262)
                  at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:107)
                  at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4797)
                  at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5473)
                  at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
                  at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
                  at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
                  at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:634)
                  at org.ofbiz.catalina.container.CatalinaContainer.configureContext(CatalinaContainer.java:766)
                  at org.ofbiz.catalina.container.CatalinaContainer.access$000(CatalinaContainer.java:147)
                  at org.ofbiz.catalina.container.CatalinaContainer$1.call(CatalinaContainer.java:617)
                  at org.ofbiz.catalina.container.CatalinaContainer$1.call(CatalinaContainer.java:615)
                  at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
                  at java.util.concurrent.FutureTask.run(FutureTask.java:166)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
                  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
                  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
                  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
                  at java.lang.Thread.run(Thread.java:724)
          Caused by: java.lang.ClassCastException: class org.apache.lucene.codecs.lucene45.Lucene45DocValuesFormat
                  at java.lang.Class.asSubclass(Class.java:3116)
                  at org.apache.lucene.util.SPIClassIterator.next(SPIClassIterator.java:137)
                  at org.apache.lucene.util.NamedSPILoader.reload(NamedSPILoader.java:65)
                  at org.apache.lucene.util.NamedSPILoader.<init>(NamedSPILoader.java:47)
                  at org.apache.lucene.util.NamedSPILoader.<init>(NamedSPILoader.java:37)
                  at org.apache.lucene.codecs.DocValuesFormat.<clinit>(DocValuesFormat.java:43)
                  ... 26 more
          
          2015-04-06 18:51:05,507 (catalina-startup-0) [ SolrDispatchFilter.java:131:INFO ] SolrDispatchFilter.init() done
          
          Show
          sanjeev Sanjeev Gupta added a comment - - edited Hi All, I'm testing this patch on trunk (a slightly outdated release 1559083 on a windows machine ) but getting the following error .. Just wondering if anyone has faced this issue or can help me fix the issue .. 2015-04-06 18:51:05,241 (catalina-startup-0) [ SolrDispatchFilter.java:116:INFO ] SolrDispatchFilter.init() 2015-04-06 18:51:05,318 (catalina-startup-0) [ SolrResourceLoader.java:683:INFO ] No /solr/home in JNDI 2015-04-06 18:51:05,318 (catalina-startup-0) [ SolrResourceLoader.java:693:INFO ] using system property solr.solr.home: specialpurpose/solr 2015-04-06 18:51:05,323 (catalina-startup-0) [ SolrResourceLoader.java:115:INFO ] new SolrResourceLoader for directory: 'specialpurpose/solr\' 2015-04-06 18:51:05,327 (catalina-startup-0) [ SolrResourceLoader.java:207:INFO ] Adding 'file:/D:/dwaCommerce/trunk-1559083/specialpurpose/solr/lib/solr-solrj-4.4.0.jar' to classloader 2015-04-06 18:51:05,426 (catalina-startup-0) [ SolrDispatchFilter.java:127:ERROR] Could not start Solr. Check solr/home property and the logs 2015-04-06 18:51:05,507 (catalina-startup-0) [ SolrException.java:119:ERROR] null :java.lang.ExceptionInInitializerError at org.apache.solr.core.SolrResourceLoader.reloadLuceneSPI(SolrResourceLoader.java:184) at org.apache.solr.core.SolrResourceLoader.<init>(SolrResourceLoader.java:121) at org.apache.solr.core.SolrResourceLoader.<init>(SolrResourceLoader.java:235) at org.apache.solr.core.CoreContainer.<init>(CoreContainer.java:129) at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:139) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:122) at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:281) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:262) at org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:107) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4797) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5473) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:634) at org.ofbiz.catalina.container.CatalinaContainer.configureContext(CatalinaContainer.java:766) at org.ofbiz.catalina.container.CatalinaContainer.access$000(CatalinaContainer.java:147) at org.ofbiz.catalina.container.CatalinaContainer$1.call(CatalinaContainer.java:617) at org.ofbiz.catalina.container.CatalinaContainer$1.call(CatalinaContainer.java:615) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:724) Caused by: java.lang.ClassCastException: class org.apache.lucene.codecs.lucene45.Lucene45DocValuesFormat at java.lang. Class .asSubclass( Class .java:3116) at org.apache.lucene.util.SPIClassIterator.next(SPIClassIterator.java:137) at org.apache.lucene.util.NamedSPILoader.reload(NamedSPILoader.java:65) at org.apache.lucene.util.NamedSPILoader.<init>(NamedSPILoader.java:47) at org.apache.lucene.util.NamedSPILoader.<init>(NamedSPILoader.java:37) at org.apache.lucene.codecs.DocValuesFormat.<clinit>(DocValuesFormat.java:43) ... 26 more 2015-04-06 18:51:05,507 (catalina-startup-0) [ SolrDispatchFilter.java:131:INFO ] SolrDispatchFilter.init() done
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I recently wanted to create a branch with this new component with an updated Solr lib because of the vulnerabilty commented above. I tried the last patch I did (with Solr 4.4.0 lib) and got the same, unfortunately I did not have enough time to fix it, so I did not create the branch. I hope to do so, help would be appreciate...

          Show
          jacques.le.roux Jacques Le Roux added a comment - I recently wanted to create a branch with this new component with an updated Solr lib because of the vulnerabilty commented above. I tried the last patch I did (with Solr 4.4.0 lib) and got the same, unfortunately I did not have enough time to fix it, so I did not create the branch. I hope to do so, help would be appreciate...
          Hide
          sanjeev Sanjeev Gupta added a comment -

          Sure .. I'd be happy to help, pls advise..
          Meantime, I did some digging around and found a couple of links suggesting that it's an issue that it's a file system access issue and has been fixed in Solr 5.0; I updated the solr-solrj-4.4.0.jar with solr-solrj-5.0.0.jar but that didn't seem to help either.

          Show
          sanjeev Sanjeev Gupta added a comment - Sure .. I'd be happy to help, pls advise.. Meantime, I did some digging around and found a couple of links suggesting that it's an issue that it's a file system access issue and has been fixed in Solr 5.0; I updated the solr-solrj-4.4.0.jar with solr-solrj-5.0.0.jar but that didn't seem to help either.
          Hide
          sumitp Sumit Pandit added a comment -

          Hi Sanjeev,
          Solr, when upgrade, it does conflict with existing lucene*, http* jars present in application/content/lib & framework/base/lib. It need to be upgrade with new jars received in solr distribution.
          Please try following, this is what I have done;

          Remove following Jars

          • applications/content/lib/lucene-core-3.5.0.jar
          • framework/base/lib/httpclient-4.1*.jar
          • framework/base/lib/httpcore-4.1*.jar
          • framework/base/lib/httpmime-4.1*.jar

          Move following files into framework\base\lib directory

          • /hot-deploy/solr/lib/solr/httpclient-4.3.jar
          • /hot-deploy/solr/lib/solr/httpcore-4.3.1.jar
          • /hot-deploy/solr/lib/solr/httpmime-4.3.1.jar

          While removing lucene from content, build would failed at content due to some deprecated methods.
          You would required to fix it, in order to do so please try following -

          Index: applications/content/src/org/ofbiz/content/search/SearchWorker.java
          ===================================================================
          --- applications/content/src/org/ofbiz/content/search/SearchWorker.java	(revision 1629808)
          +++ applications/content/src/org/ofbiz/content/search/SearchWorker.java	(working copy)
          @@ -26,6 +26,16 @@
           import javolution.util.FastList;
           import javolution.util.FastMap;
           
          +import org.apache.lucene.analysis.Analyzer;
          +import org.apache.lucene.analysis.standard.StandardAnalyzer;
          +import org.apache.lucene.document.Document;
          +import org.apache.lucene.index.IndexReader;
          +import org.apache.lucene.index.IndexWriter;
          +import org.apache.lucene.index.IndexWriterConfig;
          +import org.apache.lucene.index.Term;
          +import org.apache.lucene.store.Directory;
          +import org.apache.lucene.store.FSDirectory;
          +import org.apache.lucene.util.Version;
           import org.ofbiz.base.util.Debug;
           import org.ofbiz.base.util.UtilDateTime;
           import org.ofbiz.base.util.UtilGenerics;
          @@ -38,19 +48,8 @@
           import org.ofbiz.entity.GenericValue;
           import org.ofbiz.service.LocalDispatcher;
           
          -import org.apache.lucene.analysis.standard.StandardAnalyzer;
          -import org.apache.lucene.analysis.Analyzer;
          -import org.apache.lucene.document.Document;
          -import org.apache.lucene.index.IndexReader;
          -import org.apache.lucene.index.IndexWriter;
          -import org.apache.lucene.index.IndexWriterConfig;
          -import org.apache.lucene.index.Term;
          -import org.apache.lucene.store.Directory;
          -import org.apache.lucene.store.FSDirectory;
          -import org.apache.lucene.util.Version;
           
           
          -
           /**
            * SearchWorker Class
            */
          @@ -108,7 +107,7 @@
                   List<GenericValue> contentList = null;
                   IndexReader reader = null;
                   try {
          -            reader = IndexReader.open(directory, false);
          +            reader = IndexReader.open(directory);
                   } catch (Exception e) {
                       // ignore
                   }
          @@ -163,8 +162,8 @@
                   String contentId = content.getString("contentId");
                   Term term = new Term("contentId", contentId);
                   if (Debug.infoOn()) Debug.logInfo("in indexContent, term:" + term, module);
          -        int qtyDeleted = reader.deleteDocuments(term);
          -        if (Debug.infoOn()) Debug.logInfo("in indexContent, qtyDeleted:" + qtyDeleted, module);
          +        //int qtyDeleted = reader.deleteDocuments(term);
          +        //if (Debug.infoOn()) Debug.logInfo("in indexContent, qtyDeleted:" + qtyDeleted, module);
                   String dataResourceId = content.getString("dataResourceId");
                   if (dataResourceId != null) {
                       deleteDataResourceDocument(dataResourceId, reader);
          @@ -174,8 +173,8 @@
               public static void deleteDataResourceDocument(String dataResourceId, IndexReader reader) throws Exception {
                   Term term = new Term("dataResourceId", dataResourceId);
                   if (Debug.infoOn()) Debug.logInfo("in indexContent, term:" + term, module);
          -        int qtyDeleted = reader.deleteDocuments(term);
          -        if (Debug.infoOn()) Debug.logInfo("in indexContent, qtyDeleted:" + qtyDeleted, module);
          +        //int qtyDeleted = reader.deleteDocuments(term);
          +        //if (Debug.infoOn()) Debug.logInfo("in indexContent, qtyDeleted:" + qtyDeleted, module);
               }
           
               public static void indexContent(LocalDispatcher dispatcher, Delegator delegator, Map<String, Object> context, GenericValue content, String path) throws Exception {
          Index: applications/content/build.xml
          ===================================================================
          --- applications/content/build.xml	(revision 1629808)
          +++ applications/content/build.xml	(working copy)
          @@ -30,8 +30,9 @@
               <property name="ofbiz.home.dir" value="../.."/>
           
               <path id="local.class.path">
          -        <fileset dir="${lib.dir}" includes="*.jar"/>
          +        <fileset dir="${lib.dir}" includes="*.jar" excludes="lucene*.jar"/>
                   <fileset dir="${lib.dir}/uno" includes="*.jar"/>
          +        <fileset dir="../../hot-deploy/solr/lib/solr" includes="*.jar"/>
                   <fileset dir="../../framework/base/lib" includes="*.jar"/>
                   <fileset dir="../../framework/base/lib/commons" includes="*.jar"/>
                   <fileset dir="../../framework/base/lib/j2eespecs" includes="*.jar"/>
          @@ -53,6 +54,7 @@
                   <exclude name="org/ofbiz/content/report/JREntityListIteratorDataSource.java"/>
                   <exclude name="org/ofbiz/content/report/JRMapCollectionDataSource.java"/>
                   <exclude name="org/ofbiz/content/openoffice/*.java"/>
          +        <exclude name="org/ofbiz/content/test/*.java"/>
               </patternset>
           
               <target name="jar" depends="classes">
          
          

          Note that if any of the change doesn't applied for you, please ignore.
          I think it should fix the errors you got at your end. I'll try to help what I have done for my implementation.

          Show
          sumitp Sumit Pandit added a comment - Hi Sanjeev, Solr, when upgrade, it does conflict with existing lucene*, http* jars present in application/content/lib & framework/base/lib. It need to be upgrade with new jars received in solr distribution. Please try following, this is what I have done; Remove following Jars applications/content/lib/lucene-core-3.5.0.jar framework/base/lib/httpclient-4.1*.jar framework/base/lib/httpcore-4.1*.jar framework/base/lib/httpmime-4.1*.jar Move following files into framework\base\lib directory /hot-deploy/solr/lib/solr/httpclient-4.3.jar /hot-deploy/solr/lib/solr/httpcore-4.3.1.jar /hot-deploy/solr/lib/solr/httpmime-4.3.1.jar While removing lucene from content, build would failed at content due to some deprecated methods. You would required to fix it, in order to do so please try following - Index: applications/content/src/org/ofbiz/content/search/SearchWorker.java =================================================================== --- applications/content/src/org/ofbiz/content/search/SearchWorker.java (revision 1629808) +++ applications/content/src/org/ofbiz/content/search/SearchWorker.java (working copy) @@ -26,6 +26,16 @@ import javolution.util.FastList; import javolution.util.FastMap; + import org.apache.lucene.analysis.Analyzer; + import org.apache.lucene.analysis.standard.StandardAnalyzer; + import org.apache.lucene.document.Document; + import org.apache.lucene.index.IndexReader; + import org.apache.lucene.index.IndexWriter; + import org.apache.lucene.index.IndexWriterConfig; + import org.apache.lucene.index.Term; + import org.apache.lucene.store.Directory; + import org.apache.lucene.store.FSDirectory; + import org.apache.lucene.util.Version; import org.ofbiz.base.util.Debug; import org.ofbiz.base.util.UtilDateTime; import org.ofbiz.base.util.UtilGenerics; @@ -38,19 +48,8 @@ import org.ofbiz.entity.GenericValue; import org.ofbiz.service.LocalDispatcher; - import org.apache.lucene.analysis.standard.StandardAnalyzer; - import org.apache.lucene.analysis.Analyzer; - import org.apache.lucene.document.Document; - import org.apache.lucene.index.IndexReader; - import org.apache.lucene.index.IndexWriter; - import org.apache.lucene.index.IndexWriterConfig; - import org.apache.lucene.index.Term; - import org.apache.lucene.store.Directory; - import org.apache.lucene.store.FSDirectory; - import org.apache.lucene.util.Version; - /** * SearchWorker Class */ @@ -108,7 +107,7 @@ List<GenericValue> contentList = null ; IndexReader reader = null ; try { - reader = IndexReader.open(directory, false ); + reader = IndexReader.open(directory); } catch (Exception e) { // ignore } @@ -163,8 +162,8 @@ String contentId = content.getString( "contentId" ); Term term = new Term( "contentId" , contentId); if (Debug.infoOn()) Debug.logInfo( "in indexContent, term:" + term, module); - int qtyDeleted = reader.deleteDocuments(term); - if (Debug.infoOn()) Debug.logInfo( "in indexContent, qtyDeleted:" + qtyDeleted, module); + // int qtyDeleted = reader.deleteDocuments(term); + // if (Debug.infoOn()) Debug.logInfo( "in indexContent, qtyDeleted:" + qtyDeleted, module); String dataResourceId = content.getString( "dataResourceId" ); if (dataResourceId != null ) { deleteDataResourceDocument(dataResourceId, reader); @@ -174,8 +173,8 @@ public static void deleteDataResourceDocument( String dataResourceId, IndexReader reader) throws Exception { Term term = new Term( "dataResourceId" , dataResourceId); if (Debug.infoOn()) Debug.logInfo( "in indexContent, term:" + term, module); - int qtyDeleted = reader.deleteDocuments(term); - if (Debug.infoOn()) Debug.logInfo( "in indexContent, qtyDeleted:" + qtyDeleted, module); + // int qtyDeleted = reader.deleteDocuments(term); + // if (Debug.infoOn()) Debug.logInfo( "in indexContent, qtyDeleted:" + qtyDeleted, module); } public static void indexContent(LocalDispatcher dispatcher, Delegator delegator, Map< String , Object > context, GenericValue content, String path) throws Exception { Index: applications/content/build.xml =================================================================== --- applications/content/build.xml (revision 1629808) +++ applications/content/build.xml (working copy) @@ -30,8 +30,9 @@ <property name= "ofbiz.home.dir" value= "../.." /> <path id= "local.class.path" > - <fileset dir= "${lib.dir}" includes= "*.jar" /> + <fileset dir= "${lib.dir}" includes= "*.jar" excludes= "lucene*.jar" /> <fileset dir= "${lib.dir}/uno" includes= "*.jar" /> + <fileset dir= "../../hot-deploy/solr/lib/solr" includes= "*.jar" /> <fileset dir= "../../framework/base/lib" includes= "*.jar" /> <fileset dir= "../../framework/base/lib/commons" includes= "*.jar" /> <fileset dir= "../../framework/base/lib/j2eespecs" includes= "*.jar" /> @@ -53,6 +54,7 @@ <exclude name= "org/ofbiz/content/report/JREntityListIteratorDataSource.java" /> <exclude name= "org/ofbiz/content/report/JRMapCollectionDataSource.java" /> <exclude name= "org/ofbiz/content/openoffice/*.java" /> + <exclude name= "org/ofbiz/content/test/*.java" /> </patternset> <target name= "jar" depends= "classes" > Note that if any of the change doesn't applied for you, please ignore. I think it should fix the errors you got at your end. I'll try to help what I have done for my implementation.
          Hide
          sanjeev Sanjeev Gupta added a comment -

          Thx Sumit, Will try out the suggestion and update on the outcome..

          Show
          sanjeev Sanjeev Gupta added a comment - Thx Sumit, Will try out the suggestion and update on the outcome..
          Hide
          sanjeev Sanjeev Gupta added a comment -

          Hi Sumit,
          So you were Right .. the problem was the conflict in jar files. Thanks a lot for your guidance.
          In my case however I didn't get the compile error when I removed the jars as you mentioned ( probably might get there when I start with data indexing and query) . I did thought have a tough time making sure that all the compatible libraries were in place to get the setup right.

          Hi Jacques
          Here a few simple steps that can be follow to ensure that the Slor setup works well along with the patch provided:-
          1) Ensure that the Solr libraries included match the version of Lucene libraries included in the OFBiz release.
          2) http client libraries can be removed from the solr component
          3) Remove the old lucene libraries being used by speicalpurpose/cmssite component. Update it's references to the lucene libraries in speicalpurpose/lucene component.
          Pls let me know if you'd like me to update the patch for a specific version.

          Show
          sanjeev Sanjeev Gupta added a comment - Hi Sumit, So you were Right .. the problem was the conflict in jar files. Thanks a lot for your guidance. In my case however I didn't get the compile error when I removed the jars as you mentioned ( probably might get there when I start with data indexing and query) . I did thought have a tough time making sure that all the compatible libraries were in place to get the setup right. Hi Jacques Here a few simple steps that can be follow to ensure that the Slor setup works well along with the patch provided:- 1) Ensure that the Solr libraries included match the version of Lucene libraries included in the OFBiz release. 2) http client libraries can be removed from the solr component 3) Remove the old lucene libraries being used by speicalpurpose/cmssite component. Update it's references to the lucene libraries in speicalpurpose/lucene component. Pls let me know if you'd like me to update the patch for a specific version.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Thank you both,

          I already spotted it was a conflict with Lucene, but did not went as further as you. With this in hand I will create a branch to allow us to share and progress

          Show
          jacques.le.roux Jacques Le Roux added a comment - Thank you both, I already spotted it was a conflict with Lucene, but did not went as further as you. With this in hand I will create a branch to allow us to share and progress
          Hide
          sumitp Sumit Pandit added a comment -

          Dear All, Can anybody tell me; keeping solr in hot-deploy, how can I run the it on different port with ofbiz. As per right now both is accessible on port 8080.

          Show
          sumitp Sumit Pandit added a comment - Dear All, Can anybody tell me; keeping solr in hot-deploy, how can I run the it on different port with ofbiz. As per right now both is accessible on port 8080.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Just a bet: you could follow the way I used with the portoffset parameter https://cwiki.apache.org/confluence/display/OFBIZ/Main+New+Features#MainNewFeatures-October2013

          Show
          jacques.le.roux Jacques Le Roux added a comment - Just a bet: you could follow the way I used with the portoffset parameter https://cwiki.apache.org/confluence/display/OFBIZ/Main+New+Features#MainNewFeatures-October2013
          Hide
          sumitp Sumit Pandit added a comment -

          Dear All,
          I am using 4.10. In it getting memory leak issue. Can someone help me in fix the leak. If he has got it in past.

          Show
          sumitp Sumit Pandit added a comment - Dear All, I am using 4.10. In it getting memory leak issue. Can someone help me in fix the leak. If he has got it in past.
          Hide
          shi.jinghai Shi Jinghai added a comment -

          If no objection, I'll try to add the solr component into trunk/specialpurpose/solr in 2 or 3 weeks and upgrade to solr-4.9 match the lucene component.

          Show
          shi.jinghai Shi Jinghai added a comment - If no objection, I'll try to add the solr component into trunk/specialpurpose/solr in 2 or 3 weeks and upgrade to solr-4.9 match the lucene component.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          That sounds great Jinghai, I should be able to help if necessary...

          Show
          jacques.le.roux Jacques Le Roux added a comment - That sounds great Jinghai, I should be able to help if necessary...
          Hide
          pfm.smits Pierre Smits added a comment -

          Shi Jinghai Looking forward to it. But how will this affect/influence current implementation of the Lucene elements (e.g. in special purpose/lucene) and its use in ecommerce, contentmgr, product?

          Show
          pfm.smits Pierre Smits added a comment - Shi Jinghai Looking forward to it. But how will this affect/influence current implementation of the Lucene elements (e.g. in special purpose/lucene) and its use in ecommerce, contentmgr, product?
          Hide
          shi.jinghai Shi Jinghai added a comment -

          To Jacques:
          Thank you Jacques! I downloaded the "solr in specialpurpose.zip" and am working on upgrading it. I wish it's the latest version.

          To Pierre:
          I think the solr component will not affect current implement of Lucene elements, besides, it depends on the lucent component. In my mind, solr will be used as http/json/xml search request interface and lucene indexes management tool.

          Show
          shi.jinghai Shi Jinghai added a comment - To Jacques: Thank you Jacques! I downloaded the "solr in specialpurpose.zip" and am working on upgrading it. I wish it's the latest version. To Pierre: I think the solr component will not affect current implement of Lucene elements, besides, it depends on the lucent component. In my mind, solr will be used as http/json/xml search request interface and lucene indexes management tool.
          Hide
          jacques.le.roux Jacques Le Roux added a comment - - edited

          I just checked my comments above. I think we should at least update Solr tp 4.10.3 due to security issues. Maybe even use the last version if the current code in the ZIP can (barely?) support it?

          I mean this https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-3628

          Show
          jacques.le.roux Jacques Le Roux added a comment - - edited I just checked my comments above. I think we should at least update Solr tp 4.10.3 due to security issues. Maybe even use the last version if the current code in the ZIP can (barely?) support it? I mean this https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-3628
          Hide
          shi.jinghai Shi Jinghai added a comment -

          Thanks Jacques for the input! It's very helpful. Yes, I think the current code can support latest solr as solr api is steady. What I'll do if updated to latest solr will be to use the new solr.xml instead of the -Dsolr.solr.home config and make indexes point to runtime/indexes.

          Show
          shi.jinghai Shi Jinghai added a comment - Thanks Jacques for the input! It's very helpful. Yes, I think the current code can support latest solr as solr api is steady. What I'll do if updated to latest solr will be to use the new solr.xml instead of the -Dsolr.solr.home config and make indexes point to runtime/indexes.
          Hide
          pfm.smits Pierre Smits added a comment -

          If current (Lucene) code isn't expected to be an issue, it might be considered to upgrade all related to the latest stable version available (Lucene/Solr 5.2.1) if that proves to deliver additional benefits. Especially given that this won't affect current released versions and the next is still far away.

          Show
          pfm.smits Pierre Smits added a comment - If current (Lucene) code isn't expected to be an issue, it might be considered to upgrade all related to the latest stable version available (Lucene/Solr 5.2.1) if that proves to deliver additional benefits. Especially given that this won't affect current released versions and the next is still far away.
          Hide
          shi.jinghai Shi Jinghai added a comment -

          Thanks Pierre! I'll try to get this by 2 steps: upgrade solr to 4.9.0 and then I'll check what version both lucene and solor can reach.

          Show
          shi.jinghai Shi Jinghai added a comment - Thanks Pierre! I'll try to get this by 2 steps: upgrade solr to 4.9.0 and then I'll check what version both lucene and solor can reach.
          Hide
          pfm.smits Pierre Smits added a comment -

          I have created https://cwiki.apache.org/confluence/display/OFBIZ/Enhanced+Indexing as a descriptive page for the lucene component. When the Solr solution is available we can set up a similar page for that. Or if achievable integrate the two.

          Show
          pfm.smits Pierre Smits added a comment - I have created https://cwiki.apache.org/confluence/display/OFBIZ/Enhanced+Indexing as a descriptive page for the lucene component. When the Solr solution is available we can set up a similar page for that. Or if achievable integrate the two.
          Hide
          deepak.dixit Deepak Dixit added a comment -

          http://wiki.apache.org/solr/EmbeddedSolr
          https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+4+to+Solr+5#MajorChangesfromSolr4toSolr5-SolrisNowaStandaloneServer

          Internally, Solr is still implemented via Servlet APIs and is powered by Jetty – but this is simply an implementation detail. Deployment as a "webapp" to other Servlet Containers (or other instances of Jetty) is not supported, and may not work in future 5.x versions of Solr when additional changes are likely to be made to Solr internally to leverage custom networking stack features.

          Are we considering this as well?

          Show
          deepak.dixit Deepak Dixit added a comment - http://wiki.apache.org/solr/EmbeddedSolr https://cwiki.apache.org/confluence/display/solr/Major+Changes+from+Solr+4+to+Solr+5#MajorChangesfromSolr4toSolr5-SolrisNowaStandaloneServer Internally, Solr is still implemented via Servlet APIs and is powered by Jetty – but this is simply an implementation detail. Deployment as a "webapp" to other Servlet Containers (or other instances of Jetty) is not supported, and may not work in future 5.x versions of Solr when additional changes are likely to be made to Solr internally to leverage custom networking stack features. Are we considering this as well?
          Hide
          shi.jinghai Shi Jinghai added a comment -

          Solr is from Lucene, Lucene from Tomcat. Does it mean Solr will support his mistress Jetty only? I can't believe it. Hahahaha, but this will be my favorite topic.

          Show
          shi.jinghai Shi Jinghai added a comment - Solr is from Lucene, Lucene from Tomcat. Does it mean Solr will support his mistress Jetty only? I can't believe it. Hahahaha, but this will be my favorite topic.
          Hide
          pfm.smits Pierre Smits added a comment - - edited

          Doesn't that make the 'OFBiz' life after Solr 5.x easier. The solrj api can connect to an external Solr implementation. See https://cwiki.apache.org/confluence/display/solr/Using+SolrJ. But still there is the EmbeddedSolrServer functionality. Though the page states that particularly SolrCloud and IndexReplication aren't support, this might prove to be of an minor issue when having it embedded in OFBiz.

          Show
          pfm.smits Pierre Smits added a comment - - edited Doesn't that make the 'OFBiz' life after Solr 5.x easier. The solrj api can connect to an external Solr implementation. See https://cwiki.apache.org/confluence/display/solr/Using+SolrJ . But still there is the EmbeddedSolrServer functionality. Though the page states that particularly SolrCloud and IndexReplication aren't support, this might prove to be of an minor issue when having it embedded in OFBiz.
          Hide
          shi.jinghai Shi Jinghai added a comment -

          I submitted some code on this task into trunk, the changes are:
          1. Put the patch into trunk.
          2. Added access control to solr /admin/ pages.
          3. Upgraded solr to 4.9.0 to match Lucene version in trunk.
          4. Removed solr.solr.home, used
          SolrResourceLoader loader = new SolrResourceLoader("specialpurpose/solr/conf");
          instead.

          Test cases are:
          1. For a logged in admin user, visit http://localhost:8080/solr/, the page will be forwarded to https://localhost:8443/solr/
          2. For an anonymous use, visit http://localhost:8080/solr/, the page will be forwarded to https://localhost:8443/solr/control/checkLogin.
          3. After login, the page will be forwarded to /solr/#/.
          4. When session timeout, click on solr admin pages, the url will be forwarded to /solr/control/checkLogin.

          The files under specialpurpose/solr/webapp/solr are solr.war of solr 4.9.0. Only specialpurpose/solr/webapp/solr/js/scripts/cores.js is changed to fulfill forward admin pages to checkLogin page, the places changed are marked by "/** OFBiz customization **/" comment.

          In next phase, I'll
          1. add example(s) to use the solr.
          2. add access control to solr /update request properly.
          3. if possible, upgrade lucene and solr to the latest release.

          Show
          shi.jinghai Shi Jinghai added a comment - I submitted some code on this task into trunk, the changes are: 1. Put the patch into trunk. 2. Added access control to solr /admin/ pages. 3. Upgraded solr to 4.9.0 to match Lucene version in trunk. 4. Removed solr.solr.home, used SolrResourceLoader loader = new SolrResourceLoader("specialpurpose/solr/conf"); instead. Test cases are: 1. For a logged in admin user, visit http://localhost:8080/solr/ , the page will be forwarded to https://localhost:8443/solr/ 2. For an anonymous use, visit http://localhost:8080/solr/ , the page will be forwarded to https://localhost:8443/solr/control/checkLogin . 3. After login, the page will be forwarded to /solr/#/. 4. When session timeout, click on solr admin pages, the url will be forwarded to /solr/control/checkLogin. The files under specialpurpose/solr/webapp/solr are solr.war of solr 4.9.0. Only specialpurpose/solr/webapp/solr/js/scripts/cores.js is changed to fulfill forward admin pages to checkLogin page, the places changed are marked by "/** OFBiz customization **/" comment. In next phase, I'll 1. add example(s) to use the solr. 2. add access control to solr /update request properly. 3. if possible, upgrade lucene and solr to the latest release.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I did not review in detail but this looks good so far, thanks Jinghai

          Show
          jacques.le.roux Jacques Le Roux added a comment - I did not review in detail but this looks good so far, thanks Jinghai
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          If we want more security we need to update also Lucene in trunk, right?

          Show
          jacques.le.roux Jacques Le Roux added a comment - If we want more security we need to update also Lucene in trunk, right?
          Hide
          shi.jinghai Shi Jinghai added a comment -

          Yes, I think so. I guess what I have to do is only update Lucene jars.

          Show
          shi.jinghai Shi Jinghai added a comment - Yes, I think so. I guess what I have to do is only update Lucene jars.
          Hide
          pfm.smits Pierre Smits added a comment -

          While performing a cursory review of the notification emails regarding the incorporation of the code into trunk, I noticed a variety of licenses applied. We need to ensure that we do the right thing regarding this.

          Show
          pfm.smits Pierre Smits added a comment - While performing a cursory review of the notification emails regarding the incorporation of the code into trunk, I noticed a variety of licenses applied. We need to ensure that we do the right thing regarding this.
          Hide
          shi.jinghai Shi Jinghai added a comment -

          Thanks Pierre! I'll check the licenses of the jars and update the /LICENSE file.

          Show
          shi.jinghai Shi Jinghai added a comment - Thanks Pierre! I'll check the licenses of the jars and update the /LICENSE file.
          Hide
          pfm.smits Pierre Smits added a comment -

          While trying to build the new implementation (in a dev env. that is in sync with trunk), I got following build failure:

          classes:
            [javac17] Compiling 8 source files to /Users/pierre/dev/wrkspc-luna/ofbiz-solr/specialpurpose/solr/build/classes
            [javac17] warning: [options] bootstrap class path not set in conjunction with -source 1.7
            [javac17] /Users/pierre/dev/wrkspc-luna/ofbiz-solr/specialpurpose/solr/src/org/ofbiz/solr/ProductUtil.java:158: error: cannot access ContentWrapper
            [javac17]                 String detitle = productContentDe.get("PRODUCT_NAME").toString();
            [javac17]                                                  ^
            [javac17]   class file for org.ofbiz.content.content.ContentWrapper not found
            [javac17] /Users/pierre/dev/wrkspc-luna/ofbiz-solr/specialpurpose/solr/src/org/ofbiz/solr/webapp/OFBizSolrContextFilter.java:509: error: cannot access ResourceLoader
            [javac17]         ConfigSolr config = ConfigSolr.fromSolrHome(loader, loader.getInstanceDir());
            [javac17]                                                                   ^
            [javac17]   class file for org.apache.lucene.analysis.util.ResourceLoader not found
            [javac17] 2 errors
          
          BUILD FAILED
          
          Show
          pfm.smits Pierre Smits added a comment - While trying to build the new implementation (in a dev env. that is in sync with trunk), I got following build failure: classes: [javac17] Compiling 8 source files to /Users/pierre/dev/wrkspc-luna/ofbiz-solr/specialpurpose/solr/build/classes [javac17] warning: [options] bootstrap class path not set in conjunction with -source 1.7 [javac17] /Users/pierre/dev/wrkspc-luna/ofbiz-solr/specialpurpose/solr/src/org/ofbiz/solr/ProductUtil.java:158: error: cannot access ContentWrapper [javac17] String detitle = productContentDe.get( "PRODUCT_NAME" ).toString(); [javac17] ^ [javac17] class file for org.ofbiz.content.content.ContentWrapper not found [javac17] /Users/pierre/dev/wrkspc-luna/ofbiz-solr/specialpurpose/solr/src/org/ofbiz/solr/webapp/OFBizSolrContextFilter.java:509: error: cannot access ResourceLoader [javac17] ConfigSolr config = ConfigSolr.fromSolrHome(loader, loader.getInstanceDir()); [javac17] ^ [javac17] class file for org.apache.lucene.analysis.util.ResourceLoader not found [javac17] 2 errors BUILD FAILED
          Hide
          pfm.smits Pierre Smits added a comment -

          Accessing http://demo-trunk-ofbiz.apache.org/solr/ resulted in:

          The webpage at http://demo-trunk-ofbiz.apache.org/solr/ has resulted in too many redirects. Clearing your cookies for this site or allowing third-party cookies may fix the problem. If not, it is possibly a server configuration issue and not a problem with your computer.
          
          Show
          pfm.smits Pierre Smits added a comment - Accessing http://demo-trunk-ofbiz.apache.org/solr/ resulted in: The webpage at http: //demo-trunk-ofbiz.apache.org/solr/ has resulted in too many redirects. Clearing your cookies for this site or allowing third-party cookies may fix the problem. If not, it is possibly a server configuration issue and not a problem with your computer.
          Hide
          shi.jinghai Shi Jinghai added a comment -

          Thanks Pierre! I'll check this problem.

          Show
          shi.jinghai Shi Jinghai added a comment - Thanks Pierre! I'll check this problem.
          Hide
          shi.jinghai Shi Jinghai added a comment -

          Please check whether lucene component is enabled. Solr component depends on it.

          Show
          shi.jinghai Shi Jinghai added a comment - Please check whether lucene component is enabled. Solr component depends on it.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I have reported similar at OFBIZ-6591.

          Show
          jacques.le.roux Jacques Le Roux added a comment - I have reported similar at OFBIZ-6591 .
          Hide
          shi.jinghai Shi Jinghai added a comment -

          Yes, thanks. I think this is reason why Pierre has a build problem.

          I'll submit the patch to trunk.

          Show
          shi.jinghai Shi Jinghai added a comment - Yes, thanks. I think this is reason why Pierre has a build problem. I'll submit the patch to trunk.
          Hide
          shi.jinghai Shi Jinghai added a comment -

          Pierre,

          I have updated specialpurpse/LICENSE to include all jars under specialpurpose/solr/, trunk rev. 1700461.

          Thanks,

          Show
          shi.jinghai Shi Jinghai added a comment - Pierre, I have updated specialpurpse/LICENSE to include all jars under specialpurpose/solr/, trunk rev. 1700461. Thanks,
          Hide
          pfm.smits Pierre Smits added a comment -

          I would adjust the ofbiz-component.xml file in accordance with attached patch. This will ensure that the application is visible in the secondary top menu (with an appropriate name for the menu-item) and help drive adoption.

          Show
          pfm.smits Pierre Smits added a comment - I would adjust the ofbiz-component.xml file in accordance with attached patch. This will ensure that the application is visible in the secondary top menu (with an appropriate name for the menu-item) and help drive adoption.
          Hide
          pfm.smits Pierre Smits added a comment -

          Thanks Jacques and Shi. Works like a charm!

          Show
          pfm.smits Pierre Smits added a comment - Thanks Jacques and Shi. Works like a charm!
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Thanks Jhinghai, and thanks for the reminder Pierre!

          Show
          jacques.le.roux Jacques Le Roux added a comment - Thanks Jhinghai, and thanks for the reminder Pierre!
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I see no reasons why we should not

          Show
          jacques.le.roux Jacques Le Roux added a comment - I see no reasons why we should not
          Hide
          pfm.smits Pierre Smits added a comment -

          Now that we have the component in place and it is running ootb, we should consider creating a component placeholder in JIRA so that we can capture new JIRA issues related to the component.

          Wdyt?

          Show
          pfm.smits Pierre Smits added a comment - Now that we have the component in place and it is running ootb, we should consider creating a component placeholder in JIRA so that we can capture new JIRA issues related to the component. Wdyt?
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Done

          Show
          jacques.le.roux Jacques Le Roux added a comment - Done
          Hide
          shi.jinghai Shi Jinghai added a comment -

          Thanks Pierre! Your patch is in trunk rev. 1700599.

          Show
          shi.jinghai Shi Jinghai added a comment - Thanks Pierre! Your patch is in trunk rev. 1700599.
          Hide
          shi.jinghai Shi Jinghai added a comment -

          The ofbiz-solr-logo.png is a new logo I want to replace the solr logo. Not sure whether it's ok to go.

          Show
          shi.jinghai Shi Jinghai added a comment - The ofbiz-solr-logo.png is a new logo I want to replace the solr logo. Not sure whether it's ok to go.
          Hide
          shi.jinghai Shi Jinghai added a comment -

          I tried to login solr on demo-trunk-ofbiz and still got too many redirects problem. In the log from webtools:

          2015-09-02 04:26:19,520 |http-bio-8080-exec-6 |ConfigXMLReader |I| controller loaded: 0.0s, 4 requests, 0 views in jndi:/0.0.0.0/solr/WEB-INF/controller.xml
          2015-09-02 04:26:19,528 |http-bio-8080-exec-6 |ConfigXMLReader |I| controller loaded: 0.0s, 40 requests, 19 views in file:/home/ofbizDemo/trunk/framework/common/webcommon/WEB-INF/common-controller.xml
          2015-09-02 04:26:19,535 |http-bio-8080-exec-6 |ConfigXMLReader |I| controller loaded: 0.0s, 0 requests, 0 views in file:/home/ofbizDemo/trunk/framework/common/webcommon/WEB-INF/handlers-controller.xml
          2015-09-02 04:26:19,543 |http-bio-8080-exec-6 |ConfigXMLReader |I| controller loaded: 0.0s, 4 requests, 0 views in file:/home/ofbizDemo/trunk/applications/commonext/webapp/WEB-INF/controller.xml
          2015-09-02 04:26:20,892 |http-bio-8080-exec-9 |EntityUtilProperties |W| Could not get a system property for port.http : null
          2015-09-02 04:26:20,892 |http-bio-8080-exec-9 |EntityUtilProperties |W| Could not get a system property for force.http.host : null
          2015-09-02 04:26:20,892 |http-bio-8080-exec-9 |EntityUtilProperties |W| Could not get a system property for port.https : null
          2015-09-02 04:26:20,892 |http-bio-8080-exec-9 |EntityUtilProperties |W| Could not get a system property for force.https.host : null
          2015-09-02 04:26:20,893 |http-bio-8080-exec-9 |EntityUtilProperties |W| Could not get a system property for port.https.enabled : null
          2015-09-02 04:26:25,012 |http-bio-8080-exec-1 |ControlEventListener |I| Creating session: 737AB35A90C0CBB3761533CF15782A68.jvm1
          2015-09-02 04:26:25,013 |http-bio-8080-exec-1 |ControlServlet |T| [[[main(Domain:http://ofbiz-vm.apache.org)] Request Begun, encoding=[UTF-8]- total:0.0,since last(Begin):0.0]]
          

          I'm working on this problem.

          Show
          shi.jinghai Shi Jinghai added a comment - I tried to login solr on demo-trunk-ofbiz and still got too many redirects problem. In the log from webtools: 2015-09-02 04:26:19,520 |http-bio-8080-exec-6 |ConfigXMLReader |I| controller loaded: 0.0s, 4 requests, 0 views in jndi:/0.0.0.0/solr/WEB-INF/controller.xml 2015-09-02 04:26:19,528 |http-bio-8080-exec-6 |ConfigXMLReader |I| controller loaded: 0.0s, 40 requests, 19 views in file:/home/ofbizDemo/trunk/framework/common/webcommon/WEB-INF/common-controller.xml 2015-09-02 04:26:19,535 |http-bio-8080-exec-6 |ConfigXMLReader |I| controller loaded: 0.0s, 0 requests, 0 views in file:/home/ofbizDemo/trunk/framework/common/webcommon/WEB-INF/handlers-controller.xml 2015-09-02 04:26:19,543 |http-bio-8080-exec-6 |ConfigXMLReader |I| controller loaded: 0.0s, 4 requests, 0 views in file:/home/ofbizDemo/trunk/applications/commonext/webapp/WEB-INF/controller.xml 2015-09-02 04:26:20,892 |http-bio-8080-exec-9 |EntityUtilProperties |W| Could not get a system property for port.http : null 2015-09-02 04:26:20,892 |http-bio-8080-exec-9 |EntityUtilProperties |W| Could not get a system property for force.http.host : null 2015-09-02 04:26:20,892 |http-bio-8080-exec-9 |EntityUtilProperties |W| Could not get a system property for port.https : null 2015-09-02 04:26:20,892 |http-bio-8080-exec-9 |EntityUtilProperties |W| Could not get a system property for force.https.host : null 2015-09-02 04:26:20,893 |http-bio-8080-exec-9 |EntityUtilProperties |W| Could not get a system property for port.https.enabled : null 2015-09-02 04:26:25,012 |http-bio-8080-exec-1 |ControlEventListener |I| Creating session: 737AB35A90C0CBB3761533CF15782A68.jvm1 2015-09-02 04:26:25,013 |http-bio-8080-exec-1 |ControlServlet |T| [[[main(Domain:http: //ofbiz-vm.apache.org)] Request Begun, encoding=[UTF-8]- total:0.0,since last(Begin):0.0]] I'm working on this problem.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I like it, but 2 things

          1. moot point: it should be OFBiz not ofbiz (I know the same exists on our main site, I never liked this aspect)
          2. Not sure we can reuse the Solr Logo mixing with OFBiz name...
          Show
          jacques.le.roux Jacques Le Roux added a comment - I like it, but 2 things moot point: it should be OFBiz not ofbiz (I know the same exists on our main site, I never liked this aspect) Not sure we can reuse the Solr Logo mixing with OFBiz name...
          Hide
          pfm.smits Pierre Smits added a comment -

          Re item 2: Maybe just adding a tagline under the default OFBiz logo for the component stating: 'powered by Apache Solr' would do the trick..

          Show
          pfm.smits Pierre Smits added a comment - Re item 2: Maybe just adding a tagline under the default OFBiz logo for the component stating: 'powered by Apache Solr' would do the trick..
          Hide
          pfm.smits Pierre Smits added a comment -
          Show
          pfm.smits Pierre Smits added a comment - Or use this one (given that it is already an OFBiz application: http://svn.apache.org/repos/asf/lucene/dev/tags/lucene_solr_3_1/solr/site/images/powered_by_solr.png
          Hide
          pfm.smits Pierre Smits added a comment -

          This file contains the original description by Paul Piper.

          Show
          pfm.smits Pierre Smits added a comment - This file contains the original description by Paul Piper.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          Good idea Pierre

          Show
          jacques.le.roux Jacques Le Roux added a comment - Good idea Pierre
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I'd prefer the second

          Show
          jacques.le.roux Jacques Le Roux added a comment - I'd prefer the second
          Hide
          shi.jinghai Shi Jinghai added a comment -

          I added permission check on /update, /update/json, /update/csv and /update/extract requests in trunk rev. 1701495. These requests are defined in an index's solrconfig.xml.

          Test case steps:
          1. To make test the permission check on solr update requests easier, you can change session time out to 1 minute in specialpurpose/solr/webapp/solr/WEB-INF/web.xml.
          2. Login solr, in "Core Selector" choose "solrdefault" -> "Documents", click "Submit Document", the response of this request will be "success".
          3. Wait session timeout and click "Submit Document", the response will be "error", "Unauthorized" (http response code 401). If the user has logged in and doesn't have permission, the response will be error 403.

          Show
          shi.jinghai Shi Jinghai added a comment - I added permission check on /update, /update/json, /update/csv and /update/extract requests in trunk rev. 1701495. These requests are defined in an index's solrconfig.xml. Test case steps: 1. To make test the permission check on solr update requests easier, you can change session time out to 1 minute in specialpurpose/solr/webapp/solr/WEB-INF/web.xml. 2. Login solr, in "Core Selector" choose "solrdefault" -> "Documents", click "Submit Document", the response of this request will be "success". 3. Wait session timeout and click "Submit Document", the response will be "error", "Unauthorized" (http response code 401). If the user has logged in and doesn't have permission, the response will be error 403.
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I think we should not change how it's now

          Show
          jacques.le.roux Jacques Le Roux added a comment - I think we should not change how it's now
          Hide
          jacques.le.roux Jacques Le Roux added a comment -

          I close this issue now. We will open new issues for Solr when needed, for instance to upgrade Solr/Lucene to version 5, etc.

          Show
          jacques.le.roux Jacques Le Roux added a comment - I close this issue now. We will open new issues for Solr when needed, for instance to upgrade Solr/Lucene to version 5, etc.

            People

            • Assignee:
              jacques.le.roux Jacques Le Roux
              Reporter:
              madppiper Paul Piper
            • Votes:
              4 Vote for this issue
              Watchers:
              24 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development