Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-14768

Multipart POST is broken; a regression since 8.6

Details

    Description

      See https://www.mail-archive.com/solr-user@lucene.apache.org/msg152182.html

      The integration tests of the solarium PHP library and the integration tests of the Search API Solr Drupal module both fail on PDF extraction if executed on Solr 8.6.
      They still work on Solr 8.5.1 an earlier versions.

      2020-08-20 12:30:35.279 INFO (qtp855700733-19) [ x:5f3e6ce2810ef] o.a.s.u.p.LogUpdateProcessorFactory [5f3e6ce2810ef] webapp=/solr path=/update/extract params={json.nl=flat&commitWithin=0&omitHeader=false&resource.name=testpdf.pdf&literal.id=extract-test&commit=true&extractOnly=false&uprefix=attr_&wt=json}

      Unknown macro: {add=[extract-test (1675547519474466816)],commit=}

      0 957
      solr8_1 | 2020-08-20 12:30:35.280 WARN (qtp855700733-19) [ ] o.e.j.s.HttpChannel /solr/5f3e6ce2810ef/update/extract => java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
      solr8_1 | at org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
      solr8_1 | java.lang.NoClassDefFoundError: org/eclipse/jetty/server/MultiParts
      solr8_1 | at org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624) ~[?:?]
      solr8_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:443) ~[?:?]
      solr8_1 | at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345) ~[?:?]
      solr8_1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596) ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) ~[jetty-security-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) ~[jetty-servlet-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322) ~[jetty-rewrite-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.Server.handle(Server.java:500) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) ~[jetty-server-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) ~[jetty-util-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) ~[jetty-util-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) ~[jetty-util-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) ~[jetty-util-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) ~[jetty-util-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) ~[jetty-util-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) ~[jetty-util-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at java.lang.Thread.run(Unknown Source) [?:?]
      solr8_1 | Caused by: java.lang.ClassNotFoundException: org.eclipse.jetty.server.MultiParts
      solr8_1 | at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:565) ~[jetty-webapp-9.4.27.v20200227.jar:9.4.27.v20200227]
      solr8_1 | at java.lang.ClassLoader.loadClass(Unknown Source) ~[?:?]
      solr8_1 | ... 40 more

      Attachments

        Issue Links

          Activity

            jdoupnik Joe Doupnik added a comment -

            May I add my parallel report on the same problem.

                Where as this works with Solr v8.5.0, on v8.6.0 and 8.6.1, operations fail when Solr encounters multipart mime formatted commands used with POST. Commands are being issued by PHP v5 and v7 programs. A PHP source program snippet:

            Unable to render embedded object: File (oQy99opGysZIoiIaDYxRDickBJCSYx9 wTnzjfh4KE6POjuYu01ERHNOYYIhzO3Zlors80V3nN9YURE9NxjiHA4c97BPvfAUxhEROQEDBEOZ1Ze8xQGERE5DUOEw1nBQSpIIRB7OsEQQUREjsAQ4XBKKuslhTQrLAH vRER0RxjiCAiIqIZYYggIiKiGWGIICIiohlhiCAiIqIZYYggIiKiGfkPYYHkPAZ30LQAAAAASUVORK5CYII=) not found.

                This produces output of  Deleteindex errno=22 and the index is not deleted.

                More, when submitting files for indexing. Again error 22 is returned, and the curl response is "false".
            While debugging the root of this is handling multipart mime encoded commands and the HTTP ERROR 500 message is the tell-tale:
            First the generating program part:

            Unable to render embedded object: File (jJWyYq9UGRgAAAAASUVORK5CYII=) not found.

            Then the wireshark analysis of the exchange. A piece of the update/extract request packet showing MIME parts:

            Unable to render embedded object: File (llxONJcwv8BG TQnV9PkiIAAAAASUVORK5CYII=) not found.

            Below is what the 500 Server Error packet contains:

            Unable to render embedded object: File (dv0naZp r3ofwA0bvHrGPWbVAAAAABJRU5ErkJggg==) not found.

                            ...
                In this file submission case the file is actually submitted and indexed, but the server's response is failure with error code 22.
            Again, one key to this puzzle is that NoClassDefFoundError for MultiParts. The java code for that area differs dramatically between Solr 8.5 and 8.6.
                Thanks,
                Joe D.

            jdoupnik Joe Doupnik added a comment - May I add my parallel report on the same problem.     Where as this works with Solr v8.5.0, on v8.6.0 and 8.6.1, operations fail when Solr encounters multipart mime formatted commands used with POST. Commands are being issued by PHP v5 and v7 programs. A PHP source program snippet: Unable to render embedded object: File (oQy99opGysZIoiIaDYxRDickBJCSYx9 wTnzjfh4KE6POjuYu01ERHNOYYIhzO3Zlors80V3nN9YURE9NxjiHA4c97BPvfAUxhEROQEDBEOZ1Ze8xQGERE5DUOEw1nBQSpIIRB7OsEQQUREjsAQ4XBKKuslhTQrLAH vRER0RxjiCAiIqIZYYggIiKiGWGIICIiohlhiCAiIqIZYYggIiKiGfkPYYHkPAZ30LQAAAAASUVORK5CYII=) not found.     This produces output of  Deleteindex errno=22 and the index is not deleted.     More, when submitting files for indexing. Again error 22 is returned, and the curl response is "false". While debugging the root of this is handling multipart mime encoded commands and the HTTP ERROR 500 message is the tell-tale: First the generating program part: Unable to render embedded object: File (jJWyYq9UGRgAAAAASUVORK5CYII=) not found. Then the wireshark analysis of the exchange. A piece of the update/extract request packet showing MIME parts: Unable to render embedded object: File (llxONJcwv8BG TQnV9PkiIAAAAASUVORK5CYII=) not found. Below is what the 500 Server Error packet contains: Unable to render embedded object: File (dv0naZp r3ofwA0bvHrGPWbVAAAAABJRU5ErkJggg==) not found.                 ...     In this file submission case the file is actually submitted and indexed, but the server's response is failure with error code 22. Again, one key to this puzzle is that NoClassDefFoundError for MultiParts. The java code for that area differs dramatically between Solr 8.5 and 8.6.     Thanks,     Joe D.
            jdoupnik Joe Doupnik added a comment - - edited

            My observations were described with pictures added, thus to see them please refer to the user's list, 23 August. This include how to reproduce code snippets.

            I have added my report (from solr-users) as an email attachment to the main item.

            Joe D.

            jdoupnik Joe Doupnik added a comment - - edited My observations were described with pictures added, thus to see them please refer to the user's list, 23 August. This include how to reproduce code snippets. I have added my report (from solr-users) as an email attachment to the main item. Joe D.
            jdoupnik Joe Doupnik added a comment - - edited

            After some review of the source code of Solr 8.6.1, file SolrRequestParsers.java, and then asking Google about MultiParts, one finds that the function cleanupMultipartFiles (which is the locus of the present error) is coupled to functions (MultiPartInputStream etc) which are depreciated and even parts actively blocked. Also such a cleanup process itself is vulnerable to side effects (https://github.com/eclipse/jetty.project/issues/4383). and (https://github.com/eclipse/jetty.project/issues/4350) . These are from Nov 2019. A comment in the first reference is useful to note:

            "A simple fix was added in 9.4.26 to avoid the NPE (#4479).
            A more substantial fix using synchronization has been merged to jetty-10.0.x (#4498) which also ensures the parts are always cleaned up properly when parsing the multipart form asynchronously."

            The second reference, 4350, is has topic title of "Depreciated MultiPartInputStreamParser still used in jetty-server (MultiPartsUtilParser) but OSGi ExportPackage suppressed."

            Thus the present functions need to be replaced, and perhaps they are with jetty 10. In any case, the material is broken in Solr 8.6.x, and any fixes need adequate testing.

            Thanks,

            Joe D.

            jdoupnik Joe Doupnik added a comment - - edited After some review of the source code of Solr 8.6.1, file SolrRequestParsers.java, and then asking Google about MultiParts, one finds that the function cleanupMultipartFiles (which is the locus of the present error) is coupled to functions (MultiPartInputStream etc) which are depreciated and even parts actively blocked. Also such a cleanup process itself is vulnerable to side effects ( https://github.com/eclipse/jetty.project/issues/4383).  and ( https://github.com/eclipse/jetty.project/issues/4350 ) . These are from Nov 2019. A comment in the first reference is useful to note: "A simple fix was added in 9.4.26 to avoid the NPE ( #4479 ). A more substantial fix using synchronization has been merged to jetty-10.0.x ( #4498 ) which also ensures the parts are always cleaned up properly when parsing the multipart form asynchronously." The second reference, 4350, is has topic title of "Depreciated MultiPartInputStreamParser still used in jetty-server (MultiPartsUtilParser) but OSGi ExportPackage suppressed." Thus the present functions need to be replaced, and perhaps they are with jetty 10. In any case, the material is broken in Solr 8.6.x, and any fixes need adequate testing. Thanks, Joe D.

            Thanks Joe & Markus! Patches welcome.

            ichattopadhyaya Ishan Chattopadhyaya added a comment - Thanks Joe & Markus! Patches welcome.
            jdoupnik Joe Doupnik added a comment -

            Ishan and Markus,
                Patches. Alas, given the large untamed dense tropical jungle of
            Solr/Lucene's Java material that task is more than I can undertake
            myself. That isn't to say I have not explored and tried, but the details
            are far too diffuse and many to decode matters. As I write this I have
            yet another test build in the making, this with the problem file of
            SolrRequestParsers.java, section cleanupMultipartFiles(), replaced by a
            simple return in an attempt to avoid the not-found effect. This build
            will take quite a while to finish, assuming I gave the correct sequence
            of ant commands, and then test. Even with that the problem is not
            solved, just its use location has been verified.
                The Jetty material on the web, for example, discusses abandoning
            MultiParts in v9 to use a new, presumably faster, rendition in v10. What
            is apparent is presently there is a missing Java file, or equivalently a
            needed reference to it, to compose a working Solr. Finding such missing
            references is for those who are deeply immersed in those details.
                For easy reference I have attached the email version of my original
            report, which has details and screen captures.
                A suggestion. It is all very nice to have nearly a thousand minor
            Java "test" utilities, but the facilities end results must be tested for
            correct operation, and it is apparent that such overall testing is
            lacking presently. To help with that part my suggestion is get my
            crawler program (which exhibits the problem encountered by Markus and
            myself), strip it down to just basic core operations (create, delete,
            add a few files to) to obtain experimental results which are just the
            kind used in the field. Snippets of the PHP code are in my attached
            message, and the full material is available from https://netlab1.net, go
            to section "Presentations of long term utility", thence to "Solr/Lucene
            Search Service." Grab the offered SearchService2.1.tar.gz file which has
            PHP code and documentation, and put the crawler to work as a test tool.
            Shrinking down the crawler to just needed test essentials is easy,
            valuable and free, and can be part of the pre-release test and
            validation process.
                A second free suggestion is offer folks the complete source tar
            ball, not just parts which go out on the web to fetch lots of other
            source files. Pretend the Internet does not exist. Those "other files"
            keep changing and hence are not thoroughly tested within the full Solr
            unit. We need the complete set which has been both tested and proven
            successful, not mixing the latest bits off someone's machine. Within
            reason, if not proven correct then not shipped.
                A last freebee. I see some individual in the project wants to
            abandon Tika and friends and instead just play with Lucene. That is an
            unacceptably narrow and regressive approach to the material, and it
            strongly works directly against users of Solr in the field. The entire
            package is what we need, not end users be players assembling and
            shuffling Java parts about willy nilly. Markus has commented on the
            large number of users of his Drupal interface material for Solr.
                The build process seems as if more hours need elapse to finish the
            "test" building component, so I will stop here. Oh dear, "ant package"
            failed, saying file lucene/common-build.xml, line 2331 failed (that line
            wants $(git.exe) for gosh sakes). I am building on SUSE Leap 15.2 Linux,
            not Windows.
                 I appreciate that we are dealing with a complicated assemblage of
            material with constantly changing responsible people, and with limits on
            project resources. Thus both Markus and myself are willing to assist as
            we can, but we cannot be expected to become deeply immersed in this
            dense material. That chore for those responsible people. Thus can we
            escalate matters to those folks and see if we can rectify the problem.
                Thanks,
                Joe D.

            jdoupnik Joe Doupnik added a comment - Ishan and Markus,     Patches. Alas, given the large untamed dense tropical jungle of Solr/Lucene's Java material that task is more than I can undertake myself. That isn't to say I have not explored and tried, but the details are far too diffuse and many to decode matters. As I write this I have yet another test build in the making, this with the problem file of SolrRequestParsers.java, section cleanupMultipartFiles(), replaced by a simple return in an attempt to avoid the not-found effect. This build will take quite a while to finish, assuming I gave the correct sequence of ant commands, and then test. Even with that the problem is not solved, just its use location has been verified.     The Jetty material on the web, for example, discusses abandoning MultiParts in v9 to use a new, presumably faster, rendition in v10. What is apparent is presently there is a missing Java file, or equivalently a needed reference to it, to compose a working Solr. Finding such missing references is for those who are deeply immersed in those details.     For easy reference I have attached the email version of my original report, which has details and screen captures.     A suggestion. It is all very nice to have nearly a thousand minor Java "test" utilities, but the facilities end results must be tested for correct operation, and it is apparent that such overall testing is lacking presently. To help with that part my suggestion is get my crawler program (which exhibits the problem encountered by Markus and myself), strip it down to just basic core operations (create, delete, add a few files to) to obtain experimental results which are just the kind used in the field. Snippets of the PHP code are in my attached message, and the full material is available from https://netlab1.net , go to section "Presentations of long term utility", thence to "Solr/Lucene Search Service." Grab the offered SearchService2.1.tar.gz file which has PHP code and documentation, and put the crawler to work as a test tool. Shrinking down the crawler to just needed test essentials is easy, valuable and free, and can be part of the pre-release test and validation process.     A second free suggestion is offer folks the complete source tar ball, not just parts which go out on the web to fetch lots of other source files. Pretend the Internet does not exist. Those "other files" keep changing and hence are not thoroughly tested within the full Solr unit. We need the complete set which has been both tested and proven successful, not mixing the latest bits off someone's machine. Within reason, if not proven correct then not shipped.     A last freebee. I see some individual in the project wants to abandon Tika and friends and instead just play with Lucene. That is an unacceptably narrow and regressive approach to the material, and it strongly works directly against users of Solr in the field. The entire package is what we need, not end users be players assembling and shuffling Java parts about willy nilly. Markus has commented on the large number of users of his Drupal interface material for Solr.     The build process seems as if more hours need elapse to finish the "test" building component, so I will stop here. Oh dear, "ant package" failed, saying file lucene/common-build.xml, line 2331 failed (that line wants $(git.exe) for gosh sakes). I am building on SUSE Leap 15.2 Linux, not Windows.      I appreciate that we are dealing with a complicated assemblage of material with constantly changing responsible people, and with limits on project resources. Thus both Markus and myself are willing to assist as we can, but we cannot be expected to become deeply immersed in this dense material. That chore for those responsible people. Thus can we escalate matters to those folks and see if we can rectify the problem.     Thanks,     Joe D.
            colvinco#1 colvinco#1 added a comment - - edited

            Just to add this to the mix, while having look at this while running 8.6.2 from Eclipse with the Eclipse Jetty plugin, I hit a ClassCastException. I don't know what request prompted this to happen, but it looks like there's a couple of issues here, not just the java.lang.NoClassDefFoundError, dsmiley

            2020-09-06 17:23:18.044:WARN:oejs.HttpChannel:qtp94345706-152: /solr/main_index/update
            java.lang.ClassCastException: org.eclipse.jetty.server.MultiParts$MultiPartsUtilParser cannot be cast to org.eclipse.jetty.server.MultiParts
            	at org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624)
            	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:443)
            	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
            	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
            	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
            	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
            	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
            	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
            	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
            	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
            	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
            	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
            	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
            	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215)
            	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
            	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
            	at org.eclipse.jetty.server.Server.handle(Server.java:500)
            	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
            	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
            	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
            	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
            	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
            	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
            	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
            	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
            	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
            	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
            	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
            	at java.lang.Thread.run(Thread.java:748)

             

            colvinco#1 colvinco#1 added a comment - - edited Just to add this to the mix, while having look at this while running 8.6.2 from Eclipse with the Eclipse Jetty plugin, I hit a ClassCastException. I don't know what request prompted this to happen, but it looks like there's a couple of issues here , not just the java.lang.NoClassDefFoundError, dsmiley 2020-09-06 17:23:18.044:WARN:oejs.HttpChannel:qtp94345706-152: /solr/main_index/update java.lang.ClassCastException: org.eclipse.jetty.server.MultiParts$MultiPartsUtilParser cannot be cast to org.eclipse.jetty.server.MultiParts at org.apache.solr.servlet.SolrRequestParsers.cleanupMultipartFiles(SolrRequestParsers.java:624) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:443) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:500) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) at java.lang.Thread.run(Thread.java:748)  
            jdoupnik Joe Doupnik added a comment -

            Colvin,
                Misery loves company dept. Today Ishan Chattopadhyaya
            (jira@apache.org) finally noticed our Jira entry and asked me for
            patches. I wrote back a long reply indicating that supplying a patch was
            beyond my means though I have tried to solve the puzzle, and was in fact
            a chore for the code maintainers rather than end users. There are more
            comments in the Jetty mail archives about depreciating MultiParts and
            replacing the whole thing with new code in the v10 Jetty code (which is
            not out yet). You found even more of this. It does appear to me that
            someone either forgot to include a MultiParts file in Solr v8.6.x or
            messed up a ClassPath, or similar. Thus three editions of Solr with the
            same rather serious error. I have returned to Solr 8.5 which works
            correctly here.
                I recommended considering my exploiting my crawler program
            (https://netlab1.net, find Solr/Lucene Search Service) as one part of
            product pre-release testing because it does many common end user level
            tasks such as create, delete, add files to a core using the REST
            interface. Such tests are on the whole program, not just the internal
            thousand tests on only small components. Clearly, such whole product
            testing had not been successful.
                Just FYI, attached is my initial list message together with the
            screen capture graphics.
                Thanks,
                Joe D.

            jdoupnik Joe Doupnik added a comment - Colvin,     Misery loves company dept. Today Ishan Chattopadhyaya (jira@apache.org) finally noticed our Jira entry and asked me for patches. I wrote back a long reply indicating that supplying a patch was beyond my means though I have tried to solve the puzzle, and was in fact a chore for the code maintainers rather than end users. There are more comments in the Jetty mail archives about depreciating MultiParts and replacing the whole thing with new code in the v10 Jetty code (which is not out yet). You found even more of this. It does appear to me that someone either forgot to include a MultiParts file in Solr v8.6.x or messed up a ClassPath, or similar. Thus three editions of Solr with the same rather serious error. I have returned to Solr 8.5 which works correctly here.     I recommended considering my exploiting my crawler program ( https://netlab1.net , find Solr/Lucene Search Service) as one part of product pre-release testing because it does many common end user level tasks such as create, delete, add files to a core using the REST interface. Such tests are on the whole program, not just the internal thousand tests on only small components. Clearly, such whole product testing had not been successful.     Just FYI, attached is my initial list message together with the screen capture graphics.     Thanks,     Joe D.

            Today Ishan Chattopadhyaya (jira@apache.org) finally noticed our Jira entry and asked me for patches.

            I didn't exactly ask you, but expressed the general sentiment that this is an area where contributions will be appreciated.

            A second free suggestion

            A last freebee

            Thanks for your "free" suggestions to improve a free (and open source) project.

            in fact a chore for the code maintainers rather than end users. There

            we cannot be expected to become deeply immersed in this dense material. That chore for those responsible people. Thus can we escalate matters to those folks and see if we can rectify the problem.

            Please keep in mind that Apache Solr is a search engine, and all committers for the project are volunteers (like you) working as hard as possible to improve the project. Part of the reason why we wish to deprecate Tika/Solr Cell/Extraction is that this is one of the non-essential functionalities for our project which has not received the level and quality of support that we strive hard to provide for the rest of the project. Thank you for your suggestions, they make sense. We realize the importance of this functionality and also wish to continue supporting this functionality via non-core packages (either official Solr packages or community supported packages) going forward. Having said that, we also wish to help resolve all outstanding issues (like this) that have been caused inadvertently.

            FYI David, tallison.

            ichattopadhyaya Ishan Chattopadhyaya added a comment - Today Ishan Chattopadhyaya (jira@apache.org) finally noticed our Jira entry and asked me for patches. I didn't exactly ask you, but expressed the general sentiment that this is an area where contributions will be appreciated. A second free suggestion A last freebee Thanks for your "free" suggestions to improve a free (and open source) project. in fact a chore for the code maintainers rather than end users. There we cannot be expected to become deeply immersed in this dense material. That chore for those responsible people. Thus can we escalate matters to those folks and see if we can rectify the problem. Please keep in mind that Apache Solr is a search engine, and all committers for the project are volunteers (like you) working as hard as possible to improve the project. Part of the reason why we wish to deprecate Tika/Solr Cell/Extraction is that this is one of the non-essential functionalities for our project which has not received the level and quality of support that we strive hard to provide for the rest of the project. Thank you for your suggestions, they make sense. We realize the importance of this functionality and also wish to continue supporting this functionality via non-core packages (either official Solr packages or community supported packages) going forward. Having said that, we also wish to help resolve all outstanding issues (like this) that have been caused inadvertently. FYI David, tallison .

            Oh dear, "ant package"
            failed, saying file lucene/common-build.xml, line 2331 failed (that line
            wants $(git.exe) for gosh sakes). I am building on SUSE Leap 15.2 Linux,
            not Windows.

            Try:

            # zypper in git 

             

            ichattopadhyaya Ishan Chattopadhyaya added a comment - Oh dear, "ant package" failed, saying file lucene/common-build.xml, line 2331 failed (that line wants $(git.exe) for gosh sakes). I am building on SUSE Leap 15.2 Linux, not Windows. Try: # zypper in git  
            dsmiley David Smiley added a comment -

            I worked on the refactoring that led to this. Seems like some difficult classpath issue. Unit tests (indirectly SolrExampleXMLTest and others which randomly configure the http client to use multi-part) did not trip this and generally don't reveal classpath problems. Hmmmm... I'm investigating...

            dsmiley David Smiley added a comment - I worked on the refactoring that led to this. Seems like some difficult classpath issue. Unit tests (indirectly SolrExampleXMLTest and others which randomly configure the http client to use multi-part) did not trip this and generally don't reveal classpath problems. Hmmmm... I'm investigating...
            dsmiley David Smiley added a comment -

            I've got a fix at the PR with further info. I'm pretty happy with the fix; it uses Jetty APIs less and thus insulates us more from changes there and of course the classpath matter. I feel guilty I didn't test the multi-part stuff manually originally, and now we've got this nasty regression
            This problem is arguably worthy of a bugfix release.

            dsmiley David Smiley added a comment - I've got a fix at the PR with further info. I'm pretty happy with the fix; it uses Jetty APIs less and thus insulates us more from changes there and of course the classpath matter. I feel guilty I didn't test the multi-part stuff manually originally , and now we've got this nasty regression This problem is arguably worthy of a bugfix release.
            jdoupnik Joe Doupnik added a comment -

                git is already installed and working. Thus one suspects the Solr
            script(s) involved has(ve) problem(s). We may note that doing a "grep
            -iR git.exe *" on both solr 8.5.0 and 8.6.1 source file distributions
            produce the same results that  symbol "git.exe" appears only in the same
            lucene/common-built.xml file. Thus whatever may be the real cause of the
            8.6 test script failure it is well hidden.
                Thanks,
                Joe D.

            jdoupnik Joe Doupnik added a comment -     git is already installed and working. Thus one suspects the Solr script(s) involved has(ve) problem(s). We may note that doing a "grep -iR git.exe *" on both solr 8.5.0 and 8.6.1 source file distributions produce the same results that  symbol "git.exe" appears only in the same lucene/common-built.xml file. Thus whatever may be the real cause of the 8.6 test script failure it is well hidden.     Thanks,     Joe D.
            colvinco#1 colvinco#1 added a comment - - edited

            dsmiley thanks, I've built a new solr-core locally with the change from your PR and it resolves the problems as expected.

            colvinco#1 colvinco#1 added a comment - - edited dsmiley thanks, I've built a new solr-core locally with the change from your PR and it resolves the problems as expected.
            jdoupnik Joe Doupnik added a comment -

            Colvin and David,

            That's good news, but looking from a distance it is not yet convincing. What would help us is using regular external agents which feed files into Solr and do core create/removal as well. That is a final assembly test done in a manner understandable by others.

            I have volunteered my Linux based search service (PHP based) for this, found on https://netlab1.net/ as the Solr/Lucene Search Service. That takes less than an hour to build up completely. As its command line crawler does the work of interest here it easily blends into the needed final assembly and test operation. It's log file tells the tale.

            jdoupnik Joe Doupnik added a comment - Colvin and David, That's good news, but looking from a distance it is not yet convincing. What would help us is using regular external agents which feed files into Solr and do core create/removal as well. That is a final assembly test done in a manner understandable by others. I have volunteered my Linux based search service (PHP based) for this, found on https://netlab1.net/ as the Solr/Lucene Search Service. That takes less than an hour to build up completely. As its command line crawler does the work of interest here it easily blends into the needed final assembly and test operation. It's log file tells the tale.
            ichattopadhyaya Ishan Chattopadhyaya added a comment - - edited

            That's good news, but looking from a distance it is not yet convincing.

            Please have a closer look.

            I have volunteered my Linux based search service (PHP based) for this, found on https://netlab1.net/ as the Solr/Lucene Search Service. That takes less than an hour to build up completely. As its command line crawler does the work of interest here it easily blends into the needed final assembly and test operation.

            If you rely on Solr for this functionality and also believe your tool would take an hour to build up, why don't you set it up for your own automated testing? Alternately, feel free to open a separate JIRA/submit a patch there to help us do so.

            What would help us is using external agents ...

            Please feel free to chime in in this dev list discussion.
            https://lists.apache.org/thread.html/r27d742357f9e1342cce1c9aaa215a10d94f64b43b2e681f19fa8d150%40%3Cdev.lucene.apache.org%3E
             

            ichattopadhyaya Ishan Chattopadhyaya added a comment - - edited That's good news, but looking from a distance it is not yet convincing. Please have a closer look. I have volunteered my Linux based search service (PHP based) for this, found on https://netlab1.net/ as the Solr/Lucene Search Service. That takes less than an hour to build up completely. As its command line crawler does the work of interest here it easily blends into the needed final assembly and test operation. If you rely on Solr for this functionality and also believe your tool would take an hour to build up, why don't you set it up for your own automated testing? Alternately, feel free to open a separate JIRA/submit a patch there to help us do so. What would help us is using external agents ... Please feel free to chime in in this dev list discussion. https://lists.apache.org/thread.html/r27d742357f9e1342cce1c9aaa215a10d94f64b43b2e681f19fa8d150%40%3Cdev.lucene.apache.org%3E  
            dsmiley David Smiley added a comment -

            When I posted the PR I also started a new dev list discussion aimed at better higher level integration testing:
            https://lists.apache.org/thread.html/r27d742357f9e1342cce1c9aaa215a10d94f64b43b2e681f19fa8d150%40%3Cdev.lucene.apache.org%3E
            I think we'll be in much better shape in the upcoming month or two by adding docker-solr's tests. I'd love to see SolrExampleTests run as well because it's way more comprehensive but it will take more time. That would have caught the problem here. It ought to be a sub-task of SOLR-11872.

            dsmiley David Smiley added a comment - When I posted the PR I also started a new dev list discussion aimed at better higher level integration testing: https://lists.apache.org/thread.html/r27d742357f9e1342cce1c9aaa215a10d94f64b43b2e681f19fa8d150%40%3Cdev.lucene.apache.org%3E I think we'll be in much better shape in the upcoming month or two by adding docker-solr's tests. I'd love to see SolrExampleTests run as well because it's way more comprehensive but it will take more time. That would have caught the problem here. It ought to be a sub-task of SOLR-11872 .
            mkalkbrenner Markus Kalkbrenner added a comment - - edited

            I just want to mention that I created this issue because it has been caught by automated tests

            https://github.com/solariumphp/solarium/actions and https://github.com/mkalkbrenner/search_api_solr/actions

            Beside PDF extraction we test things like the Collections API, Streaming Expressions, Facets, JSON Facets, ...

            Therefore we run Solr docker containers. Maybe we can add a test that includes a build of a docker container using the latest Solr dev version?

            BTW both test suites allways use the latest versions of Solr 7 and 8. At the moment we had to break this rule and force Solr 8.5 instead of the latest version to get the tests to pass again to not interrupt the development of these frameworks.

            mkalkbrenner Markus Kalkbrenner added a comment - - edited I just want to mention that I created this issue because it has been caught by automated tests https://github.com/solariumphp/solarium/actions and https://github.com/mkalkbrenner/search_api_solr/actions Beside PDF extraction we test things like the Collections API, Streaming Expressions, Facets, JSON Facets, ... Therefore we run Solr docker containers. Maybe we can add a test that includes a build of a docker container using the latest Solr dev version? BTW both test suites allways use the latest versions of Solr 7 and 8. At the moment we had to break this rule and force Solr 8.5 instead of the latest version to get the tests to pass again to not interrupt the development of these frameworks.
            jdoupnik Joe Doupnik added a comment -

            Briefly responding to Ishan's most recent comment.
            The goal is produce a quality Solr for wide use, not just have fun with our own tools. One important and common element of that process is exercise a suite of pre-release tests on the entire final assemblage, in lieu of a formal beta program, and thereby try to avoid problems the day after release. The apparent lack of or deficency in that suite is at the root of this Jira thread. My offered crawler can be one such member of that suite, if the development team wishes to use it. Importantly, it is not part of the utility nor from the developers, and certainly it is not a "patch". Instead it independently represents a portion of the actual usage world. The same is true for Solarium by Markus.

            jdoupnik Joe Doupnik added a comment - Briefly responding to Ishan's most recent comment. The goal is produce a quality Solr for wide use, not just have fun with our own tools. One important and common element of that process is exercise a suite of pre-release tests on the entire final assemblage, in lieu of a formal beta program, and thereby try to avoid problems the day after release. The apparent lack of or deficency in that suite is at the root of this Jira thread. My offered crawler can be one such member of that suite, if the development team wishes to use it. Importantly, it is not part of the utility nor from the developers, and certainly it is not a "patch". Instead it independently represents a portion of the actual usage world. The same is true for Solarium by Markus.
            dsmiley David Smiley added a comment -

            A couple days ago, we finally have docker-solr integrated and are thus closer to enabling whoever to try out unreleased "dev"/snapshot releases. (Yay!) I filed SOLR-14864 for publishing nightly snapshot images. Please "Watch" that issue to be notified of activity there.

            dsmiley David Smiley added a comment - A couple days ago, we finally have docker-solr integrated and are thus closer to enabling whoever to try out unreleased "dev"/snapshot releases. (Yay!) I filed SOLR-14864 for publishing nightly snapshot images. Please "Watch" that issue to be notified of activity there.

            Commit 7b53671920840bb6fe6906436260d6bf906156f9 in lucene-solr's branch refs/heads/master from David Smiley
            [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7b53671 ]

            SOLR-14768: Fix multipart POST to Solr. (#1838)

            Regression from 8.6
            Multipart POST would fail due to a NoClassDefFoundError of Jetty MultiPart. Solr cannot access many Jetty classes, which is not noticeable in our tests.

            jira-bot ASF subversion and git services added a comment - Commit 7b53671920840bb6fe6906436260d6bf906156f9 in lucene-solr's branch refs/heads/master from David Smiley [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7b53671 ] SOLR-14768 : Fix multipart POST to Solr. (#1838) Regression from 8.6 Multipart POST would fail due to a NoClassDefFoundError of Jetty MultiPart. Solr cannot access many Jetty classes, which is not noticeable in our tests.

            Commit 08dcd24b17b6caa7bd49452dd30a66cfafbaaf89 in lucene-solr's branch refs/heads/branch_8x from David Smiley
            [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=08dcd24 ]

            SOLR-14768: Fix multipart POST to Solr. (#1838)

            Regression from 8.6
            Multipart POST would fail due to a NoClassDefFoundError of Jetty MultiPart. Solr cannot access many Jetty classes, which is not noticeable in our tests.

            (cherry picked from commit 7b53671920840bb6fe6906436260d6bf906156f9)

            jira-bot ASF subversion and git services added a comment - Commit 08dcd24b17b6caa7bd49452dd30a66cfafbaaf89 in lucene-solr's branch refs/heads/branch_8x from David Smiley [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=08dcd24 ] SOLR-14768 : Fix multipart POST to Solr. (#1838) Regression from 8.6 Multipart POST would fail due to a NoClassDefFoundError of Jetty MultiPart. Solr cannot access many Jetty classes, which is not noticeable in our tests. (cherry picked from commit 7b53671920840bb6fe6906436260d6bf906156f9)

            Commit 55a6c0ffdbba435141417bb130ffa11516a42539 in lucene-solr's branch refs/heads/branch_8_6 from David Smiley
            [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=55a6c0f ]

            SOLR-14768: Fix multipart POST to Solr. (#1838)

            Regression from 8.6
            Multipart POST would fail due to a NoClassDefFoundError of Jetty MultiPart. Solr cannot access many Jetty classes, which is not noticeable in our tests.

            (cherry picked from commit 7b53671920840bb6fe6906436260d6bf906156f9)

            jira-bot ASF subversion and git services added a comment - Commit 55a6c0ffdbba435141417bb130ffa11516a42539 in lucene-solr's branch refs/heads/branch_8_6 from David Smiley [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=55a6c0f ] SOLR-14768 : Fix multipart POST to Solr. (#1838) Regression from 8.6 Multipart POST would fail due to a NoClassDefFoundError of Jetty MultiPart. Solr cannot access many Jetty classes, which is not noticeable in our tests. (cherry picked from commit 7b53671920840bb6fe6906436260d6bf906156f9)

            Closing after the 8.6.3 release

            gerlowskija Jason Gerlowski added a comment - Closing after the 8.6.3 release

            People

              dsmiley David Smiley
              mkalkbrenner Markus Kalkbrenner
              Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m