Solr
  1. Solr
  2. SOLR-3846

TestReplicationHandler.test always (?) takes many minutes on OS X (lion)

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0-BETA, Trunk
    • Fix Version/s: 4.0, Trunk
    • Component/s: Build
    • Labels:
      None
    • Environment:

      Description

      Here's the seed was using, but this is apparently unnecessary:

      <JUnit4> says ¡Hola! Master seed: 6785BB3284A15298

      eventually it seems to complete, but it takes many minutes, for instance this was reported once, but I usually lose patience and ctrl-c out:

      [junit4:junit4] Completed on J2 in 2449.62s, 1 test
      [junit4:junit4] 
      [junit4:junit4] JVM J0:     1.21 ..   266.67 =   265.47s
      [junit4:junit4] JVM J1:     1.21 ..   238.33 =   237.12s
      [junit4:junit4] JVM J2:     1.21 ..  2538.60 =  2537.39s
      [junit4:junit4] JVM J3:     0.97 ..   267.37 =   266.40s
      [junit4:junit4] Execution time total: 42 minutes 18 seconds
      

      and a lot of lines like:
      HEARTBEAT J2: 2012-09-16T17:38:38, no events in: 187s, approx. at: TestReplicationHandler.test

      Yonik reports that he can make this happen 100% of the time on OS X/Lion, which squares with my experience as I recall. Yonik also reports...

      On my linux box (built in '09, PhenomII, HDD) the test takes 50-55 sec.
      On my kids old windows box ('08, athlon64, HDD, Win7) the test takes 88-95 sec.

      On my mac it always takes forever, and I see loops of stuff like this:

      SEVERE Master at: http://localhost:62803/solr is not available. Index
      fetch failed. Exception:
      org.apache.solr.client.solrj.SolrServerException: Server refused
      connection at: http://localhost:62803/solr
      [junit4:junit4]   2> 52751 T219 C17 UPDATE [collection1] webapp=/solr
      path=/update params={wt=javabin&version=2} {add=[150]} 0 0
      [junit4:junit4]   2> 52755 T219 C17 UPDATE [collection1] webapp=/solr
      path=/update params={wt=javabin&version=2} {add=[151]} 0 0
      [junit4:junit4]   2> 62758 T215 oash.SnapPuller.fetchLatestIndex
      SEVERE Master at: http://localhost:62803/solr is not available. Index
      fetch failed. Exception:
      

      And I'm soooo happy it's not happening to others and just being swept under the rug, restores my faith. I should have known better

      See the discussion on the dev list labeled "being a good citizen is hard when you can't successfully run tests" for more context.

      I don't know how much time I'll have to dive in to it but I'll certainly be happy to test anyone's patch.

      1. stacks.txt
        8 kB
        Mark Miller
      2. SOLR-3846.patch
        4 kB
        Uwe Schindler
      3. SOLR-3846.patch
        6 kB
        Erick Erickson
      4. SOLR-3846.patch
        19 kB
        Erick Erickson
      5. SOLR-3846.patch
        19 kB
        Erick Erickson
      6. SOLR-3846.patch
        20 kB
        Erick Erickson

        Activity

        Erick Erickson created issue -
        Erick Erickson made changes -
        Field Original Value New Value
        Description Here's the seed was using, but this is apparently unnecessary:

        <JUnit4> says ¡Hola! Master seed: 6785BB3284A15298

        _eventually_ it seems to complete, but it takes many minutes, for instance this was reported once, but I usually lose patience and ctrl-c out:
        [junit4:junit4] Completed on J2 in 2449.62s, 1 test
        [junit4:junit4]
        [junit4:junit4] JVM J0: 1.21 .. 266.67 = 265.47s
        [junit4:junit4] JVM J1: 1.21 .. 238.33 = 237.12s
        [junit4:junit4] JVM J2: 1.21 .. 2538.60 = 2537.39s
        [junit4:junit4] JVM J3: 0.97 .. 267.37 = 266.40s
        [junit4:junit4] Execution time total: 42 minutes 18 seconds

        and a lot of lines like:
        HEARTBEAT J2: 2012-09-16T17:38:38, no events in: 187s, approx. at: TestReplicationHandler.test


        Yonik reports that he can make this happen 100% of the time on OS X/Lion, which squares with my experience as I recall. Yonik also reports...

        On my linux box (built in '09, PhenomII, HDD) the test takes 50-55 sec.
        On my kids old windows box ('08, athlon64, HDD, Win7) the test takes 88-95 sec.

        On my mac it always takes forever, and I see loops of stuff like this:


        SEVERE Master at: http://localhost:62803/solr is not available. Index
        fetch failed. Exception:
        org.apache.solr.client.solrj.SolrServerException: Server refused
        connection at: http://localhost:62803/solr
        [junit4:junit4] 2> 52751 T219 C17 UPDATE [collection1] webapp=/solr
        path=/update params={wt=javabin&version=2} {add=[150]} 0 0
        [junit4:junit4] 2> 52755 T219 C17 UPDATE [collection1] webapp=/solr
        path=/update params={wt=javabin&version=2} {add=[151]} 0 0
        [junit4:junit4] 2> 62758 T215 oash.SnapPuller.fetchLatestIndex
        SEVERE Master at: http://localhost:62803/solr is not available. Index
        fetch failed. Exception:

        And I'm soooo happy it's not happening to others and just being swept under the rug, restores my faith. I should have known better ;)

        See the discussion on the dev list labeled "being a good citizen is hard when you can't successfully run tests" for more context.

        I don't know how much time I'll have to dive in to it but I'll certainly be happy to test anyone's patch.
        Here's the seed was using, but this is apparently unnecessary:

        <JUnit4> says ¡Hola! Master seed: 6785BB3284A15298

        _eventually_ it seems to complete, but it takes many minutes, for instance this was reported once, but I usually lose patience and ctrl-c out:
        {code}
        [junit4:junit4] Completed on J2 in 2449.62s, 1 test
        [junit4:junit4]
        [junit4:junit4] JVM J0: 1.21 .. 266.67 = 265.47s
        [junit4:junit4] JVM J1: 1.21 .. 238.33 = 237.12s
        [junit4:junit4] JVM J2: 1.21 .. 2538.60 = 2537.39s
        [junit4:junit4] JVM J3: 0.97 .. 267.37 = 266.40s
        [junit4:junit4] Execution time total: 42 minutes 18 seconds
        {code}
        and a lot of lines like:
        HEARTBEAT J2: 2012-09-16T17:38:38, no events in: 187s, approx. at: TestReplicationHandler.test


        Yonik reports that he can make this happen 100% of the time on OS X/Lion, which squares with my experience as I recall. Yonik also reports...

        On my linux box (built in '09, PhenomII, HDD) the test takes 50-55 sec.
        On my kids old windows box ('08, athlon64, HDD, Win7) the test takes 88-95 sec.

        On my mac it always takes forever, and I see loops of stuff like this:

        {code}
        SEVERE Master at: http://localhost:62803/solr is not available. Index
        fetch failed. Exception:
        org.apache.solr.client.solrj.SolrServerException: Server refused
        connection at: http://localhost:62803/solr
        [junit4:junit4] 2> 52751 T219 C17 UPDATE [collection1] webapp=/solr
        path=/update params={wt=javabin&version=2} {add=[150]} 0 0
        [junit4:junit4] 2> 52755 T219 C17 UPDATE [collection1] webapp=/solr
        path=/update params={wt=javabin&version=2} {add=[151]} 0 0
        [junit4:junit4] 2> 62758 T215 oash.SnapPuller.fetchLatestIndex
        SEVERE Master at: http://localhost:62803/solr is not available. Index
        fetch failed. Exception:
        {code}

        And I'm soooo happy it's not happening to others and just being swept under the rug, restores my faith. I should have known better ;)

        See the discussion on the dev list labeled "being a good citizen is hard when you can't successfully run tests" for more context.

        I don't know how much time I'll have to dive in to it but I'll certainly be happy to test anyone's patch.
        Hide
        Mark Miller added a comment -

        I'm seeing this on my macbook air as well.

        Show
        Mark Miller added a comment - I'm seeing this on my macbook air as well.
        Hide
        Mark Miller added a comment -

        Some of the more interesting stack traces where this is hanging...odd...

        Show
        Mark Miller added a comment - Some of the more interesting stack traces where this is hanging...odd...
        Mark Miller made changes -
        Attachment stacks.txt [ 12545361 ]
        Hide
        Mark Miller added a comment -

        Those threads are locked up good - stack traces in the attached stacks.txt file.

        Show
        Mark Miller added a comment - Those threads are locked up good - stack traces in the attached stacks.txt file.
        Hide
        Mark Miller added a comment -

        I suppose the most suspicious looking fellow is:

        "qtp1716711701-794" prio=5 tid=7fe8a1b03800 nid=0x10f124000 waiting for monitor entry [10f122000]
           java.lang.Thread.State: BLOCKED (on object monitor)
        	at java.security.Permissions.implies(Permissions.java:162)
        	- waiting to lock <7dee37460> (a java.security.Permissions)
        	at sun.security.provider.PolicyFile.implies(PolicyFile.java:1111)
        	at java.security.ProtectionDomain.implies(ProtectionDomain.java:224)
        	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:352)
        	at java.security.AccessController.checkPermission(AccessController.java:546)
        	at java.lang.SecurityManager.checkPermission(SecurityManager.java:532)
        	at java.lang.reflect.AccessibleObject.setAccessible(AccessibleObject.java:107)
        	at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.ConstructorConstructor.newDefaultConstructor(ConstructorConstructor.java:84)
        	at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.ConstructorConstructor.getConstructor(ConstructorConstructor.java:66)
        	at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:64)
        	at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.MiniGson.getAdapter(MiniGson.java:92)
        	at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:504)
        	at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:61)
        	- locked <7dee9ec18> (a java.lang.Object)
        	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$4.write(SlaveMain.java:329)
        	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        	- locked <7dee420a0> (a java.io.BufferedOutputStream)
        	at java.io.PrintStream.flush(PrintStream.java:288)
        	- locked <7deeaebc8> (a java.io.PrintStream)
        	at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278)
        	at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
        	- locked <7def6b868> (a java.io.OutputStreamWriter)
        	at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
        	at java.util.logging.StreamHandler.flush(StreamHandler.java:225)
        	- locked <7def6ae48> (a java.util.logging.ConsoleHandler)
        	at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:89)
        	at java.util.logging.Logger.log(Logger.java:478)
        	at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:579)
        	at org.slf4j.impl.JDK14LoggerAdapter.info(JDK14LoggerAdapter.java:276)
        	at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:210)
        	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
        
        
        Show
        Mark Miller added a comment - I suppose the most suspicious looking fellow is: "qtp1716711701-794" prio=5 tid=7fe8a1b03800 nid=0x10f124000 waiting for monitor entry [10f122000] java.lang. Thread .State: BLOCKED (on object monitor) at java.security.Permissions.implies(Permissions.java:162) - waiting to lock <7dee37460> (a java.security.Permissions) at sun.security.provider.PolicyFile.implies(PolicyFile.java:1111) at java.security.ProtectionDomain.implies(ProtectionDomain.java:224) at java.security.AccessControlContext.checkPermission(AccessControlContext.java:352) at java.security.AccessController.checkPermission(AccessController.java:546) at java.lang. SecurityManager .checkPermission( SecurityManager .java:532) at java.lang.reflect.AccessibleObject.setAccessible(AccessibleObject.java:107) at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.ConstructorConstructor.newDefaultConstructor(ConstructorConstructor.java:84) at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.ConstructorConstructor.getConstructor(ConstructorConstructor.java:66) at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.ReflectiveTypeAdapterFactory.create(ReflectiveTypeAdapterFactory.java:64) at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.internal.bind.MiniGson.getAdapter(MiniGson.java:92) at com.carrotsearch.ant.tasks.junit4.dependencies.com.google.gson.Gson.toJson(Gson.java:504) at com.carrotsearch.ant.tasks.junit4.events.Serializer.serialize(Serializer.java:61) - locked <7dee9ec18> (a java.lang. Object ) at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain$4.write(SlaveMain.java:329) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123) - locked <7dee420a0> (a java.io.BufferedOutputStream) at java.io.PrintStream.flush(PrintStream.java:288) - locked <7deeaebc8> (a java.io.PrintStream) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:278) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122) - locked <7def6b868> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212) at java.util.logging.StreamHandler.flush(StreamHandler.java:225) - locked <7def6ae48> (a java.util.logging.ConsoleHandler) at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:89) at java.util.logging.Logger.log(Logger.java:478) at org.slf4j.impl.JDK14LoggerAdapter.log(JDK14LoggerAdapter.java:579) at org.slf4j.impl.JDK14LoggerAdapter.info(JDK14LoggerAdapter.java:276) at org.apache.solr.update.processor.LogUpdateProcessor.finish(LogUpdateProcessorFactory.java:210) at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:83)
        Hide
        Uwe Schindler added a comment -

        I think the first stack trace is more valuable than the one you posted. Maybe the problem is some crazy problem with IPv6 here, thats enabled by default on OS-X. It looks like this test maybe seems to use not "127.0.0.1" but instead "localhost" or anything else like InetAddrAdress.getLocalhost() and thats slow to lookup on OS-X? Intersttingly only this test has this problem, so there must be something different in this test - the lock here comes from the AccessController that waits for resolving some IPv6 adress to check if it falls. It similar to the stupid FreeBSD blackhole, some tests were more affected by this.

        Show
        Uwe Schindler added a comment - I think the first stack trace is more valuable than the one you posted. Maybe the problem is some crazy problem with IPv6 here, thats enabled by default on OS-X. It looks like this test maybe seems to use not "127.0.0.1" but instead "localhost" or anything else like InetAddrAdress.getLocalhost() and thats slow to lookup on OS-X? Intersttingly only this test has this problem, so there must be something different in this test - the lock here comes from the AccessController that waits for resolving some IPv6 adress to check if it falls. It similar to the stupid FreeBSD blackhole, some tests were more affected by this.
        Hide
        Uwe Schindler added a comment -

        Can you try the attached patch if it brings back the test to speed?

        If this solves the issue, we should fix some other tests (5 or so, most solr tests use correctly 127.0.0.1) to use the localhost ip adress directly.

        The problem here is known to me, that resolving localhost on IPv6 enabled machines is slow and often causes problems. It looks like OS-X seems to go through DNS to get the ip adress (maybe it no longer has /etc/hosts).

        Don't ask me why this is a problem, but I was fighting against that several times (especially windows machines), so don't care and use the ip adress, we already do this in most othe tests.

        Show
        Uwe Schindler added a comment - Can you try the attached patch if it brings back the test to speed? If this solves the issue, we should fix some other tests (5 or so, most solr tests use correctly 127.0.0.1) to use the localhost ip adress directly. The problem here is known to me, that resolving localhost on IPv6 enabled machines is slow and often causes problems. It looks like OS-X seems to go through DNS to get the ip adress (maybe it no longer has /etc/hosts). Don't ask me why this is a problem, but I was fighting against that several times (especially windows machines), so don't care and use the ip adress, we already do this in most othe tests.
        Uwe Schindler made changes -
        Attachment SOLR-3846.patch [ 12545363 ]
        Hide
        Dawid Weiss added a comment -

        And I'm soooo happy it's not happening to others and just being swept under the rug, restores my faith. I should have known better

        Yes, I'm for sweeping things that don't work under the (JIRA-marked and annotated) rug, Yonik. Otherwise you get people like Erick who are on the edge of not running tests at all. I think it's bad and I think this kind of shows we're both right but have a different angle on this.

        Show
        Dawid Weiss added a comment - And I'm soooo happy it's not happening to others and just being swept under the rug, restores my faith. I should have known better Yes, I'm for sweeping things that don't work under the (JIRA-marked and annotated) rug, Yonik. Otherwise you get people like Erick who are on the edge of not running tests at all. I think it's bad and I think this kind of shows we're both right but have a different angle on this.
        Hide
        Erick Erickson added a comment - - edited

        Thanks Uwe!

        I tried your patch, but a variant of the problem seemed to continue, but note the "localhost" in the URL. So I grepped for "localhost" in *.java and there were none. So then I tried grepping for localhost in *.xml and found three configs that I changed to 127.0.0.1 and the tests ran (the XML changes are in the attached patch). I have no clue whether this is the right thing to do but you seemed to have identified the problem.

        I'll run the tests repeatedly today and report if anything pops up.

        Thanks again!

        [junit4:junit4] 2> 249902 T216 oash.SnapPuller.fetchLatestIndex SEVERE Master at: http://localhost:52274/solr is not available. Index fetch failed. Exception: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:52274/solr
        [junit4:junit4] 2> 249904 T220 C17 UPDATE [collection1] webapp=/solr path=/update params=

        {wt=javabin&version=2} {add=[230]} 0 0
        [junit4:junit4] HEARTBEAT J0: 2012-09-17T08:07:57, no events in: 248s, approx. at: TestReplicationHandler.test
        [junit4:junit4] 2> 249907 T220 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2}

        {add=[231]}

        0 0
        [junit4:junit4] 2> 259909 T216 oash.SnapPuller.fetchLatestIndex SEVERE Master at: http://localhost:52274/solr is not available. Index fetch failed. Exception: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:52274/solr
        [junit4:junit4] 2> 259911 T220 C17 UPDATE [collection1] webapp=/solr path=/update params=

        {wt=javabin&version=2}

        {add=[232]}

        0 1
        [junit4:junit4] 2> 269913 T216 oash.SnapPuller.fetchLatestIndex SEVERE Master at: http://localhost:52274/solr is not available. Index fetch failed. Exception: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:52274/solr
        [junit4:junit4] 2> 269

        Show
        Erick Erickson added a comment - - edited Thanks Uwe! I tried your patch, but a variant of the problem seemed to continue, but note the "localhost" in the URL. So I grepped for "localhost" in *.java and there were none. So then I tried grepping for localhost in *.xml and found three configs that I changed to 127.0.0.1 and the tests ran (the XML changes are in the attached patch). I have no clue whether this is the right thing to do but you seemed to have identified the problem. I'll run the tests repeatedly today and report if anything pops up. Thanks again! [junit4:junit4] 2> 249902 T216 oash.SnapPuller.fetchLatestIndex SEVERE Master at: http://localhost:52274/solr is not available. Index fetch failed. Exception: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:52274/solr [junit4:junit4] 2> 249904 T220 C17 UPDATE [collection1] webapp=/solr path=/update params= {wt=javabin&version=2} {add=[230]} 0 0 [junit4:junit4] HEARTBEAT J0: 2012-09-17T08:07:57, no events in: 248s, approx. at: TestReplicationHandler.test [junit4:junit4] 2> 249907 T220 C17 UPDATE [collection1] webapp=/solr path=/update params={wt=javabin&version=2} {add=[231]} 0 0 [junit4:junit4] 2> 259909 T216 oash.SnapPuller.fetchLatestIndex SEVERE Master at: http://localhost:52274/solr is not available. Index fetch failed. Exception: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:52274/solr [junit4:junit4] 2> 259911 T220 C17 UPDATE [collection1] webapp=/solr path=/update params= {wt=javabin&version=2} {add=[232]} 0 1 [junit4:junit4] 2> 269913 T216 oash.SnapPuller.fetchLatestIndex SEVERE Master at: http://localhost:52274/solr is not available. Index fetch failed. Exception: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://localhost:52274/solr [junit4:junit4] 2> 269
        Erick Erickson made changes -
        Attachment SOLR-3846.patch [ 12545404 ]
        Hide
        Uwe Schindler added a comment -

        For me there were some more localhosts in *.java files. If you confirm that the bug is fixed, we should change all those to 127.0.0.1 everywhere.

        I think the problem seems to be that on OS-X "localhost" maps to some other (maybe IPv6) ipadress and Jetty always only listens on IPv4 in default config. what does "ping localhost" show as address?

        Btw: You patch contaisn some unrelated changes in JFlex. Before committing this we should get rid of all localhosts.

        Show
        Uwe Schindler added a comment - For me there were some more localhosts in *.java files. If you confirm that the bug is fixed, we should change all those to 127.0.0.1 everywhere. I think the problem seems to be that on OS-X "localhost" maps to some other (maybe IPv6) ipadress and Jetty always only listens on IPv4 in default config. what does "ping localhost" show as address? Btw: You patch contaisn some unrelated changes in JFlex. Before committing this we should get rid of all localhosts.
        Hide
        Yonik Seeley added a comment -

        I think the problem seems to be that on OS-X "localhost" maps to some other (maybe IPv6) ipadress and Jetty always only listens on IPv4 in default config.

        Is this just our embedded jetty, or our stock server too? This may be something we want to adress in the jetty config since users are very likely to use "localhost".

        Show
        Yonik Seeley added a comment - I think the problem seems to be that on OS-X "localhost" maps to some other (maybe IPv6) ipadress and Jetty always only listens on IPv4 in default config. Is this just our embedded jetty, or our stock server too? This may be something we want to adress in the jetty config since users are very likely to use "localhost".
        Hide
        Uwe Schindler added a comment - - edited

        See also: http://bugs.sun.com/view_bug.do?bug_id=6230761 (Windows only9

        By default Jetty only listens on IPv4 addresses... (I tested it with ,my linux box, "netstat -an" shows only the IPv4 address in the LISTEN section). If (on OSX) localhost maps to ::1 and not 127.0.0.1, then this fails.

        In all cases we should use a consistent way to adress localhost in the tests, and for portability it should be 127.0.0.1 and not the "non-canonic" localhost.

        In general for running tests, we should also make the included Jetty also listen explicit on 127.0.0.1 and not 0.0.0.0 (all interfaces). When running tests on my windows box the firewall always complains that its opening a port to the internet... This can be enabled in the internal jetty runner for tests to not only pass a random port, but also pass 127.0.0.1 using setHost() on the connector.

        Show
        Uwe Schindler added a comment - - edited See also: http://bugs.sun.com/view_bug.do?bug_id=6230761 (Windows only9 By default Jetty only listens on IPv4 addresses... (I tested it with ,my linux box, "netstat -an" shows only the IPv4 address in the LISTEN section). If (on OSX) localhost maps to ::1 and not 127.0.0.1, then this fails. In all cases we should use a consistent way to adress localhost in the tests, and for portability it should be 127.0.0.1 and not the "non-canonic" localhost. In general for running tests, we should also make the included Jetty also listen explicit on 127.0.0.1 and not 0.0.0.0 (all interfaces). When running tests on my windows box the firewall always complains that its opening a port to the internet... This can be enabled in the internal jetty runner for tests to not only pass a random port, but also pass 127.0.0.1 using setHost() on the connector.
        Hide
        Erick Erickson added a comment -

        bq: Btw: You patch contaisn some unrelated changes in JFlex. Before committing this we should get rid of all localhosts.

        I meant to note that I never intended that patch to be committed, just putting it up for info. I suspect that there's some more generic fix available as you and Yonik have been discussing.

        Show
        Erick Erickson added a comment - bq: Btw: You patch contaisn some unrelated changes in JFlex. Before committing this we should get rid of all localhosts. I meant to note that I never intended that patch to be committed, just putting it up for info. I suspect that there's some more generic fix available as you and Yonik have been discussing.
        Hide
        Uwe Schindler added a comment - - edited

        This discussion was not meant as a more generic fix. The problem is to make it consistent through all tests. Most tests are already using 127.0.0.1, so we should use this address also in this tests. "localhost" is known to be broken on tons of systems (e.g. by modified /etc/hosts or by default pointing to ::1 in some linux distribs). Depending on OS kernel config, the internal redirection of ::1 to 127.0.0.1 done by lots of linux distribs is not really reliable.

        My above proposal was an additional suggestion, to also make the "listening" address for tests spawning jetty to be hardcoded to 127.0.0.1 (IPv4 is always available). This prevents security warnings in windows and also makes the machine running tests more protected (it's serious to open ports on public interfaces!). The important thing: If we bind the local test jettes to 127.0.0.1, the connection URL used in tests/configs must be 127.0.0.1, too.

        In general, I recommend to all people setting up local TomCats and Jetties (Solr) to explicitely use 127.0.0.1 as bind address in the server configs and always use the absolute localhost IP address in their client. This solves most problems and makes the systems much more secure, if external access is not needed. And if you have a reverse proxy in front, it is also mandatory to bind to 127.0.0.1 (security-wise).

        Show
        Uwe Schindler added a comment - - edited This discussion was not meant as a more generic fix. The problem is to make it consistent through all tests. Most tests are already using 127.0.0.1, so we should use this address also in this tests. "localhost" is known to be broken on tons of systems (e.g. by modified /etc/hosts or by default pointing to ::1 in some linux distribs). Depending on OS kernel config, the internal redirection of ::1 to 127.0.0.1 done by lots of linux distribs is not really reliable. My above proposal was an additional suggestion, to also make the "listening" address for tests spawning jetty to be hardcoded to 127.0.0.1 (IPv4 is always available). This prevents security warnings in windows and also makes the machine running tests more protected (it's serious to open ports on public interfaces!). The important thing: If we bind the local test jettes to 127.0.0.1, the connection URL used in tests/configs must be 127.0.0.1, too. In general, I recommend to all people setting up local TomCats and Jetties (Solr) to explicitely use 127.0.0.1 as bind address in the server configs and always use the absolute localhost IP address in their client. This solves most problems and makes the systems much more secure, if external access is not needed. And if you have a reverse proxy in front, it is also mandatory to bind to 127.0.0.1 (security-wise).
        Hide
        Mark Miller added a comment -

        I'm pretty sure this was affecting one of my tests on linux as well - it wouldn't hang, but it would randomly oscillate between taking 2 seconds and 2-3 minutes - the only pertinent stack trace I could get was in a socket read wait. I've been trying to figure out what's going on for a couple days now. There was no smoking gun getAddr stack trace like this though. Anyway, I checked and it had some localhost usage. I'm not home to test on my linux box, but that test is already looking better on my macbook air (last night it was also taking a long time on the air - this morning, after the change from localhost, it seems to be consistently faster).

        Show
        Mark Miller added a comment - I'm pretty sure this was affecting one of my tests on linux as well - it wouldn't hang, but it would randomly oscillate between taking 2 seconds and 2-3 minutes - the only pertinent stack trace I could get was in a socket read wait. I've been trying to figure out what's going on for a couple days now. There was no smoking gun getAddr stack trace like this though. Anyway, I checked and it had some localhost usage. I'm not home to test on my linux box, but that test is already looking better on my macbook air (last night it was also taking a long time on the air - this morning, after the change from localhost, it seems to be consistently faster).
        Erick Erickson made changes -
        Assignee Erick Erickson [ erickerickson ]
        Erick Erickson made changes -
        Attachment SOLR-3846.patch [ 12545404 ]
        Hide
        Erick Erickson added a comment -

        Looks like I mixed up this patch with the no-tab patch, so I removed it.

        Here's a new patch with the changes to the xml files as well as Uwe's changes. Running tests on 4x and trunk now and plan to commit it when I've verified that tests run unless there are objections.

        Show
        Erick Erickson added a comment - Looks like I mixed up this patch with the no-tab patch, so I removed it. Here's a new patch with the changes to the xml files as well as Uwe's changes. Running tests on 4x and trunk now and plan to commit it when I've verified that tests run unless there are objections.
        Erick Erickson made changes -
        Attachment SOLR-3846.patch [ 12545449 ]
        Hide
        Uwe Schindler added a comment -

        I am fine with that patch, but I found more "localhosts" in tests, grep throughout lucene:

        bash-4.1$ PATH=/bin find . -name '*.java' | xargs grep localhost
        ./solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestContentStreamDataSource.java:
             * on localhost at the specified port.
        File ./solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestSolrEntityProcessorEndToEnd.java:
          //rivate static final String SOLR_SOURCE_URL = "http://localhost:8983/solr";
            return "http://localhost:" + port + "/solr";
        File ./solr/core/src/java/org/apache/solr/cloud/SolrZkServer.java:
            return "localhost:" + zkProps.getClientPortAddress().getPort();
          // Given zkHost=localhost:1111,localhost:2222 this will inject
          // server.0=localhost:1112:1113
          // server.1=localhost:2223:2224
            String myHost = "localhost";
              // default to localhost:<solrPort+1001>
        File ./solr/core/src/java/org/apache/solr/handler/admin/ShowFileRequestHandler.java:
         *   http://localhost:8983/solr/admin/file?file=schema.xml&contentType=text/plain
        File ./solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java:
            // maps "localhost:8983|localhost:7574" to a shuffled List("http://localhost:8983","http://localhost:7574")
            // for back compat, a shards param with URLs like localhost:8983/solr will mean that this
        File ./solr/core/src/java/org/apache/solr/handler/component/ShardResponse.java:
          /** What was the shard address that returned this response.  Example:  "http://localhost:8983/solr" */
        File ./solr/core/src/java/org/apache/solr/schema/RandomSortField.java:
         * <li>http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_1234%20desc</li>
         * <li>http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_2345%20desc</li>
         * <li>http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_ABDC%20desc</li>
         * <li>http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_21%20desc</li>
        File ./solr/core/src/java/org/apache/solr/util/SimplePostTool.java:
          private static final String DEFAULT_POST_URL = "http://localhost:8983/solr/update";
             "  java -Durl=http://localhost:8983/solr/update/extract -Dparams=literal.id=a -Dtype=application/pdf -jar post.jar a.pdf\n"+
        File ./solr/core/src/test/org/apache/solr/AnalysisAfterCoreReloadTest.java:
              String url = "http://localhost:"+port+context+"/"+name;
        File ./solr/core/src/test/org/apache/solr/schema/TestBinaryField.java:
            String url = "http://localhost:" + jetty.getLocalPort() + context;
        File ./solr/core/src/test/org/apache/solr/search/TestSolrJ.java:
            String addr = "http://localhost:8983/solr";
            HttpSolrServer client = new HttpSolrServer("http://localhost:8983/solr");
        File ./solr/core/src/test/org/apache/solr/TestSolrCoreProperties.java:
            String url = "http://localhost:" + solrJetty.getLocalPort() + "/solr";
           * on localhost at the specified port.
        File ./solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpSolrServer.java:
           *          <code>http://localhost:8983/solr/</code>" if you are using the
        File ./solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttpSolrServer.java:
          // keys to the maps are currently of the form "http://localhost:8983/solr"
        File ./solr/solrj/src/test/org/apache/solr/client/solrj/embedded/JettyWebappTest.java:
            String adminPath = "http://localhost:"+port+context+"/";
        File ./solr/solrj/src/test/org/apache/solr/client/solrj/embedded/MultiCoreExampleJettyTest.java:
              String url = "http://localhost:"+port+context+"/"+name;
            String baseURL = "localhost:"+port+context+"/";
        File ./solr/solrj/src/test/org/apache/solr/client/solrj/embedded/SolrExampleJettyTest.java:
              String url = "http://localhost/?core=xxx";
        File ./solr/solrj/src/test/org/apache/solr/client/solrj/embedded/SolrExampleStreamingTest.java:
              String url = "http://localhost:"+port+context;       // smaller queue size hits locks more often
        File ./solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleBinaryTest.java:
              String url = "http://localhost:"+port+context;
        File ./solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleXMLTest.java:
              String url = "http://localhost:" + port + context;
        File ./solr/solrj/src/test/org/apache/solr/client/solrj/TestLBHttpSolrServer.java:
              return "http://localhost:" + port + "/solr";
        File ./solr/test-framework/src/java/org/apache/solr/BaseDistributedSearchTestCase.java:
          // protected String[] deadServers = {"does_not_exist_54321.com:33331/solr","localhost:33332/solr"};
              String url = "http://localhost:" + port + context;
        File ./solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java:
              sb.append("localhost:").append(j.getLocalPort()).append(context);
        File ./solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java:
              sb.append("localhost:").append(j.getLocalPort()).append(context);
              sb.append("|localhost:").append(j2.getLocalPort()).append(context);
              String url = "http://localhost:" + port + context + "/"
        File ./solr/test-framework/src/java/org/apache/solr/SolrJettyTestBase.java:
                String url = "http://localhost:"+port+context;
        bash-4.1$
        

        I think we should fix all of those in the same way, to be consistent in tests!

        Show
        Uwe Schindler added a comment - I am fine with that patch, but I found more "localhosts" in tests, grep throughout lucene: bash-4.1$ PATH=/bin find . -name '*.java' | xargs grep localhost ./solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestContentStreamDataSource.java: * on localhost at the specified port. File ./solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestSolrEntityProcessorEndToEnd.java: //rivate static final String SOLR_SOURCE_URL = "http://localhost:8983/solr"; return "http://localhost:" + port + "/solr"; File ./solr/core/src/java/org/apache/solr/cloud/SolrZkServer.java: return "localhost:" + zkProps.getClientPortAddress().getPort(); // Given zkHost=localhost:1111,localhost:2222 this will inject // server.0=localhost:1112:1113 // server.1=localhost:2223:2224 String myHost = "localhost"; // default to localhost:<solrPort+1001> File ./solr/core/src/java/org/apache/solr/handler/admin/ShowFileRequestHandler.java: * http://localhost:8983/solr/admin/file?file=schema.xml&contentType=text/plain File ./solr/core/src/java/org/apache/solr/handler/component/HttpShardHandler.java: // maps "localhost:8983|localhost:7574" to a shuffled List("http://localhost:8983","http://localhost:7574") // for back compat, a shards param with URLs like localhost:8983/solr will mean that this File ./solr/core/src/java/org/apache/solr/handler/component/ShardResponse.java: /** What was the shard address that returned this response. Example: "http://localhost:8983/solr" */ File ./solr/core/src/java/org/apache/solr/schema/RandomSortField.java: * <li>http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_1234%20desc</li> * <li>http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_2345%20desc</li> * <li>http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_ABDC%20desc</li> * <li>http://localhost:8983/solr/select/?q=*:*&fl=name&sort=rand_21%20desc</li> File ./solr/core/src/java/org/apache/solr/util/SimplePostTool.java: private static final String DEFAULT_POST_URL = "http://localhost:8983/solr/update"; " java -Durl=http://localhost:8983/solr/update/extract -Dparams=literal.id=a -Dtype=application/pdf -jar post.jar a.pdf\n"+ File ./solr/core/src/test/org/apache/solr/AnalysisAfterCoreReloadTest.java: String url = "http://localhost:"+port+context+"/"+name; File ./solr/core/src/test/org/apache/solr/schema/TestBinaryField.java: String url = "http://localhost:" + jetty.getLocalPort() + context; File ./solr/core/src/test/org/apache/solr/search/TestSolrJ.java: String addr = "http://localhost:8983/solr"; HttpSolrServer client = new HttpSolrServer("http://localhost:8983/solr"); File ./solr/core/src/test/org/apache/solr/TestSolrCoreProperties.java: String url = "http://localhost:" + solrJetty.getLocalPort() + "/solr"; * on localhost at the specified port. File ./solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpSolrServer.java: * <code>http://localhost:8983/solr/</code>" if you are using the File ./solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttpSolrServer.java: // keys to the maps are currently of the form "http://localhost:8983/solr" File ./solr/solrj/src/test/org/apache/solr/client/solrj/embedded/JettyWebappTest.java: String adminPath = "http://localhost:"+port+context+"/"; File ./solr/solrj/src/test/org/apache/solr/client/solrj/embedded/MultiCoreExampleJettyTest.java: String url = "http://localhost:"+port+context+"/"+name; String baseURL = "localhost:"+port+context+"/"; File ./solr/solrj/src/test/org/apache/solr/client/solrj/embedded/SolrExampleJettyTest.java: String url = "http://localhost/?core=xxx"; File ./solr/solrj/src/test/org/apache/solr/client/solrj/embedded/SolrExampleStreamingTest.java: String url = "http://localhost:"+port+context; // smaller queue size hits locks more often File ./solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleBinaryTest.java: String url = "http://localhost:"+port+context; File ./solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleXMLTest.java: String url = "http://localhost:" + port + context; File ./solr/solrj/src/test/org/apache/solr/client/solrj/TestLBHttpSolrServer.java: return "http://localhost:" + port + "/solr"; File ./solr/test-framework/src/java/org/apache/solr/BaseDistributedSearchTestCase.java: // protected String[] deadServers = {"does_not_exist_54321.com:33331/solr","localhost:33332/solr"}; String url = "http://localhost:" + port + context; File ./solr/test-framework/src/java/org/apache/solr/cloud/AbstractDistribZkTestBase.java: sb.append("localhost:").append(j.getLocalPort()).append(context); File ./solr/test-framework/src/java/org/apache/solr/cloud/AbstractFullDistribZkTestBase.java: sb.append("localhost:").append(j.getLocalPort()).append(context); sb.append("|localhost:").append(j2.getLocalPort()).append(context); String url = "http://localhost:" + port + context + "/" File ./solr/test-framework/src/java/org/apache/solr/SolrJettyTestBase.java: String url = "http://localhost:"+port+context; bash-4.1$ I think we should fix all of those in the same way, to be consistent in tests!
        Hide
        Yonik Seeley added a comment -

        I'm not so sure we should change non-test uses of "localhost" at this point, esp since a release is right around the corner.

        Show
        Yonik Seeley added a comment - I'm not so sure we should change non-test uses of "localhost" at this point, esp since a release is right around the corner.
        Hide
        Uwe Schindler added a comment - - edited

        I posted the whole grep, but only wanted to refer to src/test folder. I assumed it was clear from the corresponding comment.

        Show
        Uwe Schindler added a comment - - edited I posted the whole grep, but only wanted to refer to src/test folder. I assumed it was clear from the corresponding comment.
        Hide
        Erick Erickson added a comment -

        Sure, I can try that.

        Uwe:
        Is the ::1 pattern also a Bad Thing? See BaseDistributedTestCase[90] for one case... I'll grep for others if so.

        Yeah, a number of the non-test cases occur in the SolrCloud stuff. I wouldn't dare change that until consulting with people who've done significant work in that area.

        Running tests now with all the "localhost" entries I found in test files changed.....

        Show
        Erick Erickson added a comment - Sure, I can try that. Uwe: Is the ::1 pattern also a Bad Thing? See BaseDistributedTestCase [90] for one case... I'll grep for others if so. Yeah, a number of the non-test cases occur in the SolrCloud stuff. I wouldn't dare change that until consulting with people who've done significant work in that area. Running tests now with all the "localhost" entries I found in test files changed.....
        Hide
        Erick Erickson added a comment -

        OK, this patch includes substituting 127.0.0.1 for all occurrences of "localhost" in code in test files, including a bunch of base classes.

        All the tests for 4x ran, running trunk now (it's a miracle, the patch is sufficient and complete for both!).

        Sound like I should commit later tonight?

        Show
        Erick Erickson added a comment - OK, this patch includes substituting 127.0.0.1 for all occurrences of "localhost" in code in test files, including a bunch of base classes. All the tests for 4x ran, running trunk now (it's a miracle, the patch is sufficient and complete for both!). Sound like I should commit later tonight?
        Erick Erickson made changes -
        Attachment SOLR-3846.patch [ 12545460 ]
        Hide
        Uwe Schindler added a comment -

        I think the 2 IPv6 cases are sepcial: deadServers should explicitely assume not working server. I think using the IPv6 Adress was done to work around the FreeBSD blackhole (maybe). I am a little bit nervous, what happens if there is actually something listening on that port?

        File ./solr/solrj/src/test/org/apache/solr/client/solrj/SolrExceptionTest.java:
              SolrServer client = new HttpSolrServer("http://[ff01::114]:11235/solr/", httpClient);
        File ./solr/test-framework/src/java/org/apache/solr/BaseDistributedSearchTestCase.java:
          protected String[] deadServers = {"[::1]:33332/solr"};
        

        I think the one in BaseDistributedSearchTestCase should use the same fake address like the first one: ff01::114, ff01 are "reserved" private addresses in IPv6 for thing like broadcasting. The connection will fail 100%. Can you try that, too?

        Show
        Uwe Schindler added a comment - I think the 2 IPv6 cases are sepcial: deadServers should explicitely assume not working server. I think using the IPv6 Adress was done to work around the FreeBSD blackhole (maybe). I am a little bit nervous, what happens if there is actually something listening on that port? File ./solr/solrj/src/test/org/apache/solr/client/solrj/SolrExceptionTest.java: SolrServer client = new HttpSolrServer("http://[ff01::114]:11235/solr/", httpClient); File ./solr/test-framework/src/java/org/apache/solr/BaseDistributedSearchTestCase.java: protected String[] deadServers = {"[::1]:33332/solr"}; I think the one in BaseDistributedSearchTestCase should use the same fake address like the first one: ff01::114, ff01 are "reserved" private addresses in IPv6 for thing like broadcasting. The connection will fail 100%. Can you try that, too?
        Hide
        Uwe Schindler added a comment -

        Erick: I think you can commit that later. I would only like to see a test when you replace the ::1 in BaseDistributedSearchTestCase by ff01::114? The test is invalid, as Port 33332 might be used by some other service at the moment, so this is not "safe". You may also add more invalid addresses here: Maybe 3 of them, just replace the last 114 by 2 other numbers? Then you can emulate the old comment that failed on FreeBSD blackhole with 3 example servers!

        Show
        Uwe Schindler added a comment - Erick: I think you can commit that later. I would only like to see a test when you replace the ::1 in BaseDistributedSearchTestCase by ff01::114? The test is invalid, as Port 33332 might be used by some other service at the moment, so this is not "safe". You may also add more invalid addresses here: Maybe 3 of them, just replace the last 114 by 2 other numbers? Then you can emulate the old comment that failed on FreeBSD blackhole with 3 example servers!
        Hide
        Erick Erickson added a comment -

        New patch with Uwe's suggestions. All tests pass 4.x and trunk.

        When I said in my earlier message that I'd grepped java files for localhost I must have been on drugs. Or I had the filter set to *.xml or something; thanks for not believing me <G>...

        To Commit or not to Commit? That is the question. I admit I'm ignorant as an apple about why this works, but it sure makes my problem go away.

        Show
        Erick Erickson added a comment - New patch with Uwe's suggestions. All tests pass 4.x and trunk. When I said in my earlier message that I'd grepped java files for localhost I must have been on drugs. Or I had the filter set to *.xml or something; thanks for not believing me <G>... To Commit or not to Commit? That is the question. I admit I'm ignorant as an apple about why this works, but it sure makes my problem go away.
        Erick Erickson made changes -
        Attachment SOLR-3846.patch [ 12545497 ]
        Hide
        Erick Erickson added a comment -

        <hits self on head with blunt object. Repeatedly>

        Of course the damn test runs. I managed to commit the 4.x version of TestReplicationHandler with the offending test @Ignore'd.

        But I'm still reasonably optimistic, since I managed to remove it from the trunk and that runs.

        Re-running all that rot now, will post corrected patch when it completes.

        Show
        Erick Erickson added a comment - <hits self on head with blunt object. Repeatedly> Of course the damn test runs. I managed to commit the 4.x version of TestReplicationHandler with the offending test @Ignore'd. But I'm still reasonably optimistic, since I managed to remove it from the trunk and that runs. Re-running all that rot now, will post corrected patch when it completes.
        Hide
        Erick Erickson added a comment -

        This time fer sure.

        I think it's ready to commit.

        Show
        Erick Erickson added a comment - This time fer sure. I think it's ready to commit.
        Erick Erickson made changes -
        Attachment SOLR-3846.patch [ 12545513 ]
        Hide
        Uwe Schindler added a comment -

        +1, It is a great improvement! Thanks for testing. I also ran all tests on windows locally and they successfully passed!

        For another issue an improvement: In my dreams last night I found a very good solution for the problem with timeouting connections to IP addresses of "dead" servers (like we have seen in the above 2 tests). My idea how to solve this completely predictive (means it works on every computer, although firewall settings may delay execution,...):

        We already have a security manager and policy. The idea would be to also implement checkConnect() and checkResolve() for this custom manager, which is called on every connect. This method checks for a set of "default dead servers" (would work with ip adresses, or fake host names): If it gets called with such an address, it could throw IOException (like a SocketConnectException) so make the fail predcicatable. The underlying O/S's network layer is then never called, so no timeouts can occur. As SecurityManager cannot throw checked exceptions, I would use:

        Rethrow.rethrow(new IOException("Emulated network failure"));

        inside the SecurityManager.

        Show
        Uwe Schindler added a comment - +1, It is a great improvement! Thanks for testing. I also ran all tests on windows locally and they successfully passed! For another issue an improvement: In my dreams last night I found a very good solution for the problem with timeouting connections to IP addresses of "dead" servers (like we have seen in the above 2 tests). My idea how to solve this completely predictive (means it works on every computer, although firewall settings may delay execution,...): We already have a security manager and policy. The idea would be to also implement checkConnect() and checkResolve() for this custom manager, which is called on every connect. This method checks for a set of "default dead servers" (would work with ip adresses, or fake host names): If it gets called with such an address, it could throw IOException (like a SocketConnectException) so make the fail predcicatable. The underlying O/S's network layer is then never called, so no timeouts can occur. As SecurityManager cannot throw checked exceptions, I would use: Rethrow.rethrow( new IOException( "Emulated network failure" )); inside the SecurityManager.
        Hide
        Uwe Schindler added a comment -

        The blackhole problem could also be solved: If all started Jetties "register" themselves at LTC (like the MocjDirWrappers), this security manager could also keep track of used port numbers and make the no longer actoive ones fail early!

        Show
        Uwe Schindler added a comment - The blackhole problem could also be solved: If all started Jetties "register" themselves at LTC (like the MocjDirWrappers), this security manager could also keep track of used port numbers and make the no longer actoive ones fail early!
        Hide
        Erick Erickson added a comment -

        trunk: r1387099
        4.x : r1387098

        Uwe:
        Want to open another JIRA for your brainstorm?

        Show
        Erick Erickson added a comment - trunk: r1387099 4.x : r1387098 Uwe: Want to open another JIRA for your brainstorm?
        Erick Erickson made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Erick Erickson
        http://svn.apache.org/viewvc?view=revision&revision=1387098

        Fixes for SOLR-3846 (very long tests on OS X for TestReplicationHandler)

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Erick Erickson http://svn.apache.org/viewvc?view=revision&revision=1387098 Fixes for SOLR-3846 (very long tests on OS X for TestReplicationHandler)
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.
        Uwe Schindler made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Erick Erickson
            Reporter:
            Erick Erickson
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development