Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.2
    • Component/s: update
    • Labels:
      None

      Description

      It would be good to include an up-to-date jetty version for the example.

      1. start.jar
        16 kB
        Ryan McKinley
      2. lib.zip
        5.76 MB
        Ryan McKinley
      3. Jetty6.config.patch
        60 kB
        Ryan McKinley
      4. jetty-6.3-example.zip
        7.19 MB
        Ryan McKinley

        Activity

        Hide
        Ryan McKinley added a comment -

        will be included in solr1.2

        Show
        Ryan McKinley added a comment - will be included in solr1.2
        Hide
        Yonik Seeley added a comment -

        I think the issue with persistent connections on linux is due to Nagle's algorithm.
        The python client sends the HTTP headers and body separately, thus triggering it.
        The following little program below writes it all at once and gets very good performance:

        ---------mysock.py------
        import socket

        headers='''POST /solr/select HTTP/1.1
        Host: localhost:8983
        Accept-Encoding: identity
        Content-Length: 11
        Content-Type: application/x-www-form-urlencoded; charset=utf-8

        '''

        body='q=id%3A1234'
        msg = headers + body

        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect(("localhost", 8983))

        for i in xrange(10000):
        s.send(msg)
        rsp = s.recv(8192) #pray we get the complete response in one go

        Show
        Yonik Seeley added a comment - I think the issue with persistent connections on linux is due to Nagle's algorithm. The python client sends the HTTP headers and body separately, thus triggering it. The following little program below writes it all at once and gets very good performance: --------- mysock.py ------ import socket headers='''POST /solr/select HTTP/1.1 Host: localhost:8983 Accept-Encoding: identity Content-Length: 11 Content-Type: application/x-www-form-urlencoded; charset=utf-8 ''' body='q=id%3A1234' msg = headers + body s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(("localhost", 8983)) for i in xrange(10000): s.send(msg) rsp = s.recv(8192) #pray we get the complete response in one go
        Hide
        Yonik Seeley added a comment -

        Hmmm, some bad interaction with persistent connections and POST (which solr.py uses)?
        I switched the method to use GET, and it finished 10k connections in 6.3sec

        Many browsers (incorrectly) send a CR+LF after the post body. The python httplib does not do this... I wonder if that's the issue (the server waiting for the possible CR+LF so it can throw it away?) The delay seems to be about 40ms

        Show
        Yonik Seeley added a comment - Hmmm, some bad interaction with persistent connections and POST (which solr.py uses)? I switched the method to use GET, and it finished 10k connections in 6.3sec Many browsers (incorrectly) send a CR+LF after the post body. The python httplib does not do this... I wonder if that's the issue (the server waiting for the possible CR+LF so it can throw it away?) The delay seems to be about 40ms
        Hide
        Yonik Seeley added a comment -

        On that Opteron Linux box, almost identical numbers for Resin 3.0 pro, so it's not Jetty specific.

        Show
        Yonik Seeley added a comment - On that Opteron Linux box, almost identical numbers for Resin 3.0 pro, so it's not Jetty specific.
        Hide
        Yonik Seeley added a comment -

        The 10,000 single-client connection test on a dual Opteron, RHEL4 64 bit, Java5 64 bit -server
        Jetty 6.1.3, NIO connector, persistent=414s, non-persistent=11.3s
        Jetty 6.1.3, non-NIO connector, persistent=408s, non-persistent=10.2s

        Situation reversed from WinXP! The persistent connections are horribly slow.

        Show
        Yonik Seeley added a comment - The 10,000 single-client connection test on a dual Opteron, RHEL4 64 bit, Java5 64 bit -server Jetty 6.1.3, NIO connector, persistent=414s, non-persistent=11.3s Jetty 6.1.3, non-NIO connector, persistent=408s, non-persistent=10.2s Situation reversed from WinXP! The persistent connections are horribly slow.
        Hide
        Yonik Seeley added a comment -

        Switching to the non-NIO connector in Jetty 6.1.3, I get
        18.2s for persistent connections, 35s for non-persistent (again WinXP, Java5 -server)

        Show
        Yonik Seeley added a comment - Switching to the non-NIO connector in Jetty 6.1.3, I get 18.2s for persistent connections, 35s for non-persistent (again WinXP, Java5 -server)
        Hide
        Yonik Seeley added a comment -

        Doing some quick connection handling tests with the python client:
        from solr import *
        c = SolrConnection(host='localhost:8080', persistent=False)
        for i in xrange(10000):
        c.search(q='id:1234'

        Java 1.5 -server on WindowsXP

        Jetty 5.1.??, persistent=20.9s, non-persistent=44s, sometimes "address already in use" exception
        Jetty 6.1.3, persistent=20.1s, non-persistent=114s
        Tomcat 6.0.13, persistent=20.2s, non-persistent=29s, sometimes "address already in use" exception

        Could be a python client issue, (SO_REUSE_ADDR for python http-lib?), but it does seem like Jetty 6.1.3 is slower at handling new connections? Anyone have benchmarks from a different client?

        Show
        Yonik Seeley added a comment - Doing some quick connection handling tests with the python client: from solr import * c = SolrConnection(host='localhost:8080', persistent=False) for i in xrange(10000): c.search(q='id:1234' Java 1.5 -server on WindowsXP Jetty 5.1.??, persistent=20.9s, non-persistent=44s, sometimes "address already in use" exception Jetty 6.1.3, persistent=20.1s, non-persistent=114s Tomcat 6.0.13, persistent=20.2s, non-persistent=29s, sometimes "address already in use" exception Could be a python client issue, (SO_REUSE_ADDR for python http-lib?), but it does seem like Jetty 6.1.3 is slower at handling new connections? Anyone have benchmarks from a different client?
        Hide
        Yonik Seeley added a comment -

        Yeah, I saw that hadoop issue too, which is why I was planning on some quick
        indexing & querying benchmarks.

        Show
        Yonik Seeley added a comment - Yeah, I saw that hadoop issue too, which is why I was planning on some quick indexing & querying benchmarks.
        Hide
        Otis Gospodnetic added a comment -

        I'm using 6.1.3 in a pre-production project.
        I do see Hadoop guys hit a problem - see JETTY-345, which mentions 6.1.4rc1

        Show
        Otis Gospodnetic added a comment - I'm using 6.1.3 in a pre-production project. I do see Hadoop guys hit a problem - see JETTY-345, which mentions 6.1.4rc1
        Hide
        Ryan McKinley added a comment -

        applied to rev541050

        If anything goes wrong, we need to revert /example/ to rev 541049 and check NOTICE.txt and CHANGES.txt

        Show
        Ryan McKinley added a comment - applied to rev541050 If anything goes wrong, we need to revert /example/ to rev 541049 and check NOTICE.txt and CHANGES.txt
        Hide
        Yonik Seeley added a comment -

        OK, let's quickly go ahead then. There have been some JSP issues with our current Jetty version anyway.

        I'd like to have all the core changes done in the next few days so we can get a release out by the end of this month (1 week away).

        Show
        Yonik Seeley added a comment - OK, let's quickly go ahead then. There have been some JSP issues with our current Jetty version anyway. I'd like to have all the core changes done in the next few days so we can get a release out by the end of this month (1 week away).
        Hide
        Ryan McKinley added a comment -

        It has been working fine for me, but I have not been using it in production or under heavy load. Otis?

        If 1.2 is more then a week away, I think we should include it soon and take it out if anything looks fishy...

        Show
        Ryan McKinley added a comment - It has been working fine for me, but I have not been using it in production or under heavy load. Otis? If 1.2 is more then a week away, I think we should include it soon and take it out if anything looks fishy...
        Hide
        Yonik Seeley added a comment -

        Any opinions on if we should put in Jetty 6.1.3 now, or wait until after Solr 1.2 is released?

        Show
        Yonik Seeley added a comment - Any opinions on if we should put in Jetty 6.1.3 now, or wait until after Solr 1.2 is released?
        Hide
        Ryan McKinley added a comment -

        here is a zip with the example directory as we would (maybe) want it.

        This uses jetty-6.1.3

        Show
        Ryan McKinley added a comment - here is a zip with the example directory as we would (maybe) want it. This uses jetty-6.1.3
        Hide
        Otis Gospodnetic added a comment -

        I think we want to wait with the newest Jetty. I'm on the jetty ML, and the beast is not yet fully ironed out., not yet stable enough. I'd let it simmer a little longer while Greg, Jan and other Jetty folks get the deamons out.

        Show
        Otis Gospodnetic added a comment - I think we want to wait with the newest Jetty. I'm on the jetty ML, and the beast is not yet fully ironed out., not yet stable enough. I'd let it simmer a little longer while Greg, Jan and other Jetty folks get the deamons out.
        Hide
        Ryan McKinley added a comment -

        I'm not suggesting we drop JSP now... just that we definitely should not be using 2.1 specific stuff and where possible xml+xslt with a RequestHandler (SOLR-58) in the future.

        I just tried this with 6.1.1 and it works fine (with the jsp-2.0 or jsp-2.1)

        What is the best way for me to post this patch? Most of the work is adding and deleting jar files. It has two xml config files etc/jetty.xml and etc/webdefault.xml

        Show
        Ryan McKinley added a comment - I'm not suggesting we drop JSP now ... just that we definitely should not be using 2.1 specific stuff and where possible xml+xslt with a RequestHandler ( SOLR-58 ) in the future. I just tried this with 6.1.1 and it works fine (with the jsp-2.0 or jsp-2.1) What is the best way for me to post this patch? Most of the work is adding and deleting jar files. It has two xml config files etc/jetty.xml and etc/webdefault.xml
        Hide
        Yonik Seeley added a comment -

        The disad to getting rid of JSPs is it would make it much harder for someone to easily improve the look or functionality of the admin pages.

        Show
        Yonik Seeley added a comment - The disad to getting rid of JSPs is it would make it much harder for someone to easily improve the look or functionality of the admin pages.
        Hide
        Yonik Seeley added a comment -

        When you go to the download link:
        http://dist.codehaus.org/jetty/

        Show
        Yonik Seeley added a comment - When you go to the download link: http://dist.codehaus.org/jetty/
        Hide
        Ryan McKinley added a comment -

        Ideally we could replace existing JSP with servlets or SolrRequestHandlers. I don't see anything that could not be handled using a SolrRequestHandler and default wt=xslt. For an embedded solr, it would be nice not to need the JSP dependencies. This would have the added bennefit of returning XML/JSON/ruby/etc for the admin interface.

        Jetty 6.1.1? where is it? Its not on: http://docs.codehaus.org/display/JETTY/Downloading+and+Installing#download

        Show
        Ryan McKinley added a comment - Ideally we could replace existing JSP with servlets or SolrRequestHandlers. I don't see anything that could not be handled using a SolrRequestHandler and default wt=xslt. For an embedded solr, it would be nice not to need the JSP dependencies. This would have the added bennefit of returning XML/JSON/ruby/etc for the admin interface. Jetty 6.1.1? where is it? Its not on: http://docs.codehaus.org/display/JETTY/Downloading+and+Installing#download
        Hide
        Yonik Seeley added a comment -

        BTW, there's a Jetty 6.1.1 out...

        Show
        Yonik Seeley added a comment - BTW, there's a Jetty 6.1.1 out...
        Hide
        Yonik Seeley added a comment -

        We could probably ditch the JMX stuff as we don't currently use it... or does Jetty export some cool stuff via JMX that we might want to point people at for debugging?

        If we go with JSP 2.1, we should just remember not to use any 2.1 specific features yet... most containers aren't quite there yet. The latest stable version of Tomcat 5.5.20 is only on Servlet 2.4/JSP 2.0

        Show
        Yonik Seeley added a comment - We could probably ditch the JMX stuff as we don't currently use it... or does Jetty export some cool stuff via JMX that we might want to point people at for debugging? If we go with JSP 2.1, we should just remember not to use any 2.1 specific features yet... most containers aren't quite there yet. The latest stable version of Tomcat 5.5.20 is only on Servlet 2.4/JSP 2.0
        Hide
        Ryan McKinley added a comment -

        I tried running the example with jetty 6.1. In less then 5 mins, I had everything working smoothly. The only issue I see is the JSP jar files are huge (~4MB)

        The current example folder includes settings and jars for JMX, is that necessary?

        Show
        Ryan McKinley added a comment - I tried running the example with jetty 6.1. In less then 5 mins, I had everything working smoothly. The only issue I see is the JSP jar files are huge (~4MB) The current example folder includes settings and jars for JMX, is that necessary?

          People

          • Assignee:
            Ryan McKinley
            Reporter:
            Ryan McKinley
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development