Stable 9.0.3.v20130506

I changed the jetty version in a branch_4x checkout to 9.0.6.v20130930 ... this resulted in compilation errors. The API isn't the same, so this won't be a trivial patch.

Fixing the API issues looks like it's going to be more than just changing class and/or method names. I spent a few minutes on it before my morning commute and concluded that I wasn't going to be able to do it quickly, and without a lot of research, I won't know if I'm doing it correctly.

doesnt jetty 9 require java7?

Robert Muir, you are correct. I didn't even think to look at that. So this will be a trunk change only.

Is this now worth a review, since we are moving to Java 7? For 4.9?

We should make this change soon: Jetty 9 has builtin support for disabling protocols (POODLE), e.g. (from http://stackoverflow.com/questions/19913403/how-do-i-disallow-particular-ssl-protocols-in-jetty):

<Arg name="sslContextFactory">
    ...
    <Set name="excludeProtocols">
      <Array type="java.lang.String">
        <Item>SSLv3</Item>
      </Array>
     </Set>

Patch to upgrade to Jetty 9.

Some core tests still fail that I need to dig in:

   [junit4] Tests with failures:
   [junit4]   - org.apache.solr.rest.schema.TestSchemaResource (suite)
   [junit4]   - org.apache.solr.schema.TestCloudSchemaless.testDistribSearch
   [junit4]   - org.apache.solr.rest.schema.TestDynamicFieldResource (suite)
   [junit4]   - org.apache.solr.schema.TestCloudSchemaless (suite)

The jetty.xml file is going to need to change:

Prior to Jetty 9, the type of the connector specified both the protocol and the implementation used (for example, selector-based non blocking I/O vs blocking I/O, or SSL connector vs non-SSL connector). Jetty 9 has only a selector-based non blocking I/O connector, and a collection of ConnectionFactories now configure the protocol on the connector.

http://www.eclipse.org/jetty/documentation/current/configuring-connectors.html#jetty-connectors

I'm attaching this patch as a checkpoint.

1. Fixed some compile failures in StartSolrJetty and SolrRequestParserTest
2. Modified jetty.xml and solr-jetty-context.xml to conform to Jetty 9 changes

Todo:

1. Fix the SSL example
2. Fix the failing tests
3. The example doesn't work right now because unlike previous versions of jetty, jetty.home is resolved as the location of start.jar which gets resolved to the ivy cache path which messes up our configuration.
4. Jetty 9 has this concept of modules which can be configured via property files and/or by xml. We could do away with XML configuration if we want. Opinions?

Jetty 9 has this concept of modules which can be configured via property files and/or by xml. We could do away with XML configuration if we want.

I actually like the module concept but I'm not sure how much of that you are going to want to bundle in Solr itself (copying module files).

Let me know if you want a hand with any of the TODOs.

I actually like the module concept but I'm not sure how much of that you are going to want to bundle in Solr itself (copying module files).

Thanks Steve.

I think we can:

1. Bundle http, https, ssl and server modules
2. Have a start.ini which has all the properties of the server (SSL and non-SSL)
3. Enable only the http related part but have all the SSL related properties there as comments

Then I hope to have bin/solr enable/disable modules as necessary so that enabling ssl can be a matter of setting some environment variables and executing bin/solr with a special argument.

Right now we use restlet-jee which is not compatible with Jetty9 but restlet-jse 2.3 is compatible. According to Use account "steve_rowe" instead there is no special reason to use restlet-jee instead of restlet-jse so we may be able to switch to jse and upgrade the version to make it work.

I'll attach another patch as a checkpoint in an hour or so.

I made some progress getting the modules in place. The example doesn't come up yet. The classpath's correct, I think, but it cannot find the context. I'll look at this again tomorrow.

1. Solr comes up fine
2. Jetty logging works but somehow the rootLogger for /solr is always null, I am not sure why
3. I added -Djetty.home=$SOLR_SERVER_DIR in bin/solr otherwise jetty resolves jetty.home as the path of start.jar in ivy2 cache.

Now I will try to upgrade restlet to 2.3 and see if we can get all tests to pass.

Jetty logging works but somehow the rootLogger for /solr is always null, I am not sure why

Actually the same thing happens on branch_5x and it's fine. The logging works as expected.

I traced the test failures to double initialization of SolrDispatchFilter in JettySolrRunner. The root.addServlet also calls filter init for all filter added till that point. This was leading to thread leaks and test failures. I am not sure if this is a bug in jetty but adding SolrDispatchFilter after all other filters and servlets have been added works around this issue.

Now I will try to upgrade restlet to 2.3

This patch passes all the schema related tests that were failing earlier. I see HttpPartitionTest fail with this patch. The failure can be reproduced with:

ant test -Dtestcase=HttpPartitionTest -Dtests.method=testDistribSearch -Dtests.seed=B85FF6DC513799E3 -Dtests.slow=true -Dtests.locale=es_GT -Dtests.timezone=Asia/Chungking -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

I can use some help debugging the tests which use SocketProxy viz. HttpPartitionTest, ReplicationFactorTest, LeaderFailoverAfterPartitionTest and LeaderInitiatedRecoveryOnCommitTest. They do not need any specific seeds but seem to hang on almost every run.

I find that this happens after a few open/close cycles for the SocketProxy. A request after this point never reaches the Solr Jetty and the SocketProxy thread is found to be waiting on accept() (with no trace of a bridge ever being executed) as if the request never made it to the SocketProxy.

All other Solr tests pass with the last patch attached on this issue.

Still digging into the HttpPartitionTest hangs, but for me, it looks like something related to deleting the collections using the collections admin API at the end of the sub-tests. If I comment out the delete collection request (several places in the code), HttpPartitionTest works fine:

    try {
      CollectionAdminRequest.Delete req = new CollectionAdminRequest.Delete();
      req.setCollectionName(testCollectionName);
      //req.process(cloudClient);
    } catch (Exception e) {
      // don't fail the test
      log.warn("Could not delete collection {} after test completed", testCollectionName);
    }

Yeah, it is interesting that it fails right at that point. But there are other tests like SimpleCollectionCreateDeleteTest which do not fail. I changed the number of docs from 1000 to 200 and then it failed at a different point.

The interesting thing is that I added logging in CoreAdminHandler.unload and that log lines are never printed. The call to unload one of those cores is never received by Solr.

I just wanted to mention, that the move to Jetty 9 may need us to disable solr test on FreeBSD Jenkins completely. Currently, FreeBSD Jenkins uses the special sysprop "tests.jettyConnector" (or like that) to use the simple Java IO based connector.

Please also remove all references to this sysprop from build files (ANT/Maven), too!

My initial diagnosis was sort of a red herring ... it's a race condition in the SocketProxy which kindly goes away when I turn on debug logging ...

Shalin Shekhar Mangar can you try the patch I provided to SOLR-6874 to see if it fixes the issue for you here? I beast'd the HttpPartitionTest with the patched applied ontop of the latest patch here and it passed 15 of 15. Hoping you see the same.

Latest patch which includes SOLR-6874. All tests pass.

I think we should commit this and then work on a SSL switch in the bin scripts as well as easy to modify https/ssl configuration in the jetty configs.

This patch removes the tests.jettyConnector sysprop as Uwe mentioned earlier.

I just wanted to mention, that the move to Jetty 9 may need us to disable solr test on FreeBSD Jenkins completely. Currently, FreeBSD Jenkins uses the special sysprop "tests.jettyConnector" (or like that) to use the simple Java IO based connector.

I don't think we have a choice. I'd like to take advantage of async http for inter-shard communication so we need Jetty 9. Also, the FreeBSD Jenkins has been very unreliable in the past compared to your Policeman Jenkins box. I'm never quite sure whether a test failing on FreeBSD is a genuine or not.

Can you please disable the FreeBSD jenkins Uwe Schindler?

I will disable the tests of course once this is committed. Lucene trunk's tests are already disabled, because Java 8 crushes while running Solr tests (also inside networking code).

Final patch. I'll commit shortly.

Commit 1649552 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1649552 ]

SOLR-4839: Upgrade to Jetty 9

Commit 1649571 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1649571 ]

SOLR-4839: Removing extra license text from jetty xml and module files

Commit 1649584 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1649584 ]

SOLR-4839: Remove jetty.port from start.ini and add default inside jetty-http.xml

We still need to test this on windows. Timothy Potter has kindly offered to take care of it. I also see that the solr-port-console.log file is always empty after these changes but the solr.log has all the right logs.

Commit 1649689 from Timothy Potter in branch 'dev/trunk'
[ https://svn.apache.org/r1649689 ]

SOLR-4839: start Jetty 9 correctly on Windows

I committed some updates to solr.cmd for this. The odd thing is that I had to set java.io.tmpdir on Windows to get it to work as it looks like Jetty 9 is creating a file, such as startBIGNUMBER.properties, in the tmp location. I couldn't see if that was configurable (or what was making Jetty write that start.properties file out).

Eclipse was complaining for me with these @Override annotations, attached the patch to remove them.

Eclipse was complaining for me with these @Override annotations, attached the patch to remove them.

What was the complain? Maybe you aren't using Java 8 on trunk?

Anyone else having trouble running jetty tests from eclipse now?

What was the complain? Maybe you aren't using Java 8 on trunk?

I see the same thing, and everything is set to and using Java 8.

It says the super type methods being overriden do not even exist.

For some reasin we bring in this orbit thing:

    <dependency org="org.eclipse.jetty.orbit" name="javax.servlet" rev="${/org.eclipse.jetty.orbit/javax.servlet}" conf="servlet">
      <artifact name="javax.servlet" type="orbit" ext="jar"/>
    </dependency>

And that seems to bring in:

/org.eclipse.jetty.orbit/javax.servlet = 3.0.0.v201112011016

which conflicts with the javax.servlet brought in by this jetty update. Or something along those lines.

I don't know why we have this orbit thing, but I don't think we should have both these dependencies.

Looks like the replicator module uses this obit one. We need to harmonize them somehow so that eclipse ide dev is not broken.

Something like the attatched patch will work I think.

Commit 1650169 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1650169 ]

SOLR-4839: Remove dependency to jetty.orbit

Fixed, thanks Mark!

Mark Miller, the cloudera repo's copy of the new restlet servlet jar (upgraded by this issue) is instead some kind of error html page: http://repository.cloudera.com/artifactory/repo/org/restlet/jee/org.restlet.ext.servlet/2.3.0/org.restlet.ext.servlet-2.3.0.jar - can you get somebody to clean that up? The workaround in the meantime is to comment out the cloudera maven repo in lucene/ivy-settings.xml, do a resolve to download the proper artifact from the restlet maven repo, then un-comment it.

Erik Hatcher told me offline, and I confirmed, that the directory solr/server/solr-webapp/ was being deleted when Solr is run from a trunk checkout. In solr/server/contexts/solr-jetty-context.xml, we tell Jetty to use this directory as its tempDirectory.

Chris Hostetter (Unused) told me offline that he suspected that Jetty 9's new option to persist its temp directory (persistTempDirectory), which defaults to false, was causing this problem: when Jetty shuts down gracefully and this option is not set to true, its temp dir is deleted.

I verified that this is the case: after ant example and bin/solr start, solr/server/solr-webapp/ holds the exploded war, but after bin/solr stop, solr/server/solr-webapp ceases to exist.

When I add the following inside the <Configure> tag in solr-jetty-context.xml, the temp dir and the exploded war contained within are left intact after bin/solr stop:

  <Set name="persistTempDirectory">true</Set>

I think the above addition is the way to go.

Alternatively, to preserve the solr/server/solr-webapp/ directory, we could tell Jetty to use a sub-directory, but leave the temp dir persistence option at false - I tested this and it worked - when I changed the tempDirectory setting to include a sub-directory named tempdir/ as shown below, the sub-directory was created at startup and deleted at shutdown, leaving the parent directory intact:

  <Set name="tempDirectory"><Property name="jetty.base" default="."/>/solr-webapp/tempdir</Set>

If there are no objections, I'll add the persistTempDir=true setting to solr-jetty-context.xml tomorrow.

Currently ant example (soon to be renamed to ant server or maybe removed entirely - see SOLR-6926) deletes the exploded war from server/solr-webapp/ (while leaving the parent dir intact). I was curious whether this is necessary - perhaps Jetty is smart enough to do (recursive) timestamp comparison, re-exploding the war if its timestamp is newer than any of its exploded contents.

TL;DR: Jetty apparently is not smart enough to re-explode the war when it's newer than its previously exploded contents, so we should continue to purge the exploded war as part of ant example (or ant server or whatever we end up doing to build the war).

Here's how I tested:

I set persistTempDirectory=true (see above), then:

ant clean example
bin/solr start
bin/solr stop
cp -r server/solr-webapp{,.save}
ant clean example               # purges the exploded war from server/solr-webapp/
rmdir server/solr-webapp
cp -r server/solr-webapp{.save,}
bin/solr start
bin/solr stop
diff -r server/solr-webapp*

diff's output was empty (no differences) - so when the war is newer than the exploded contents, Jetty does not re-explode the war.

To confirm that if Jetty had exploded the war, there would be difference, I did the following (without first running ant example):

rm -rf server/solr-webapp/*
bin/solr start
bin/solr stop
diff -r server/solr-webapp*

This time, there were differences in three files:

diff -r server/solr-webapp/webapp/META-INF/MANIFEST.MF server/solr-webapp.save/webapp/META-INF/MANIFEST.MF
4,5c4,5
< Implementation-Version: 6.0.0-SNAPSHOT 1650174 - sarowe - 2015-01-07 2
<  0:24:25
---
> Implementation-Version: 6.0.0-SNAPSHOT 1650174 - sarowe - 2015-01-07 1
>  8:33:45
Binary files server/solr-webapp/webapp/WEB-INF/lib/solr-core-6.0.0-SNAPSHOT.jar and server/solr-webapp.save/webapp/WEB-INF/lib/solr-core-6.0.0-SNAPSHOT.jar differ
Binary files server/solr-webapp/webapp/WEB-INF/lib/solr-solrj-6.0.0-SNAPSHOT.jar and server/solr-webapp.save/webapp/WEB-INF/lib/solr-solrj-6.0.0-SNAPSHOT.jar differ

please, run precommit before committing.

Commit 1650293 from Robert Muir in branch 'dev/trunk'
[ https://svn.apache.org/r1650293 ]

SOLR-4839: fix licensing metadata. PLEASE RUN PRECOMMIT BEFORE COMMITTING.

Commit 1650295 from Robert Muir in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1650295 ]

SOLR-4839: fix licensing metadata. PLEASE RUN PRECOMMIT BEFORE COMMITTING.

Commit 1650328 from Use account "steve_rowe" instead in branch 'dev/trunk'
[ https://svn.apache.org/r1650328 ]

SOLR-4839: Set Jetty 9's new option to persist its temp directory (persistTempDirectory) - where the war is exploded - to true, so that the server/solr-webapp/ directory doesn't get deleted when Jetty shuts down gracefully

Commit 1650335 from Use account "steve_rowe" instead in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1650335 ]

SOLR-4839: Set Jetty 9's new option to persist its temp directory (persistTempDirectory) - where the war is exploded - to true, so that the server/solr-webapp/ directory doesn't get deleted when Jetty shuts down gracefully (merged trunk r1650328)

Mark Miller, the cloudera repo's copy of the new restlet servlet jar

It should not be pulling this from Cloudera, but from restlet. Something seems like it went wrong with the restlet repo and so its finding this in the cloudera repo. Or perhaps the resolvers need to be reordered. Cloudera is just for Kite until it's on Maven central, not for restlet.

Commit 1650368 from Mark Miller in branch 'dev/trunk'
[ https://svn.apache.org/r1650368 ]

SOLR-4839: Order Cloudera resolver after Sonatype and Restlet.

Commit 1650369 from Mark Miller in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1650369 ]

SOLR-4839: Order Cloudera resolver after Sonatype and Restlet.

Mark Miller] should we also remove the cloudera resolver and only keep the releases.cloudera.com one? My logic is:

• We only need the cloudera repo for kite
• Kite is present in the release repo
• The non-release repo has a bunch of third party stuff (like restlet) that may cause problems, as indicated by Steve above (the link seems to be working now, though)

Maybe it doesn't matter now that you've re-ordered the repos, but removing the cloudera resolver seems to work.

Order Cloudera resolver after Sonatype and Restlet.

It's funny - this is one of the first things I tried yesterday and it did not seem to help. Perhaps some initial user error somehow. Today it seems to change things for me. It also worked to add a force=true to the restlet repo, but that is also super slow and silly.

should we also remove the cloudera resolver and only keep the releases.cloudera.com one?

If we only need one, let's only have one. I'm not sure the genesis of both, other than that is what was in our initial downstream Cloudera Search repo. Would be great to remove one if it has popular 3rd party libs in it and we don't need it. Certainly helped confuse debugging this issue.

Commit 1650381 from Mark Miller in branch 'dev/trunk'
[ https://svn.apache.org/r1650381 ]

SOLR-4839: Remove unnecessary Cloudera resolver, leave Cloudera Releases resolver.

Commit 1650382 from Mark Miller in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1650382 ]

SOLR-4839: Remove unnecessary Cloudera resolver, leave Cloudera Releases resolver.

Commit 1650384 from Use account "steve_rowe" instead in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1650384 ]

SOLR-4839: revert persistTempDirectory=true setting in solr-jetty-context.xml - Jetty 9 isn't on 5.x yet

Mark Miller: on march 14th, you switched the Cloudera releases resolver from https to http to work around an expired SSL certificate.
I think the issue is resolved in the mean time so it could be switched back to https?

It's funny - this is one of the first things I tried yesterday and it did not seem to help. Perhaps some initial user error somehow. Today it seems to change things for me. It also worked to add a force=true to the restlet repo, but that is also super slow and silly.

I found out: You need to nuke the IVY cache, then it works. The strange thing I noticed was: Ivy was complaining about a "missing cloudera resolver", although nothing in our build files referred to it. I think, Ivy downloaded the stale metadata from the broken cloudera server and kept it. Of course later tries to download the JAR files still failed, because of stale metadata.

Does the config and doc for SSL in jetty.xml still work fine with Jetty 9?

I found that this broke the 'TestMiniSolrCloudCluster' test (or anything that doesn't specify a 'jetty.testMode' system property). If a test doesn't specify the 'jetty.testMode' property a null pointer exception is thrown by jetty because a ServerConnector is attempting to be created with a null Server. I attached a patch to fix the specific issue, though I'm not quite sure why we need to branch the code - couldn't we consolidate the two?

Are we going with Jetty 9.2?

https://webtide.com/jetty-9-2-0-released/

so it could be switched back to https?

What's the motivation? I would prefer it as well, but as we don't currently use https for maven central (I think it's a relatively new, free option now?) I did not really worry about it individually.

I'm seeing a number of new test fails when trying to stop and then start a jetty instance:

java.net.BindException: Address already in use

That failure is on 5x too and Jetty 9 is only on trunk. Has to be something else that's tripping those.

Do you have some fail links to point to?

I just started seeing this locally, and I have not seen it locally in a very long time. I'm only running trunk, but if you have some 5x fail links, I can compare what I'm seeing.

I think you commented on this one: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/11431/
Haven't looked at this at all but here Jetty doesn't start up due to a BindException. (also, has a comment from you)

Anyone happen to see the same issue I was getting with the TestMiniSolrCloudCluster? I attached a patch but doesn't look like it was pulled in yet, just wondering if others were getting similar test failures.

Commit 1657495 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1657495 ]

SOLR-4839: Avoid NPE when jetty.testMode=false

Hi Steve Davids, I have committed your patch. Thanks for catching it.

Steve Davids, do you know why you saw that but we didn't see TestMiniSolrCloudCluster fails?

Seems like we don't have any user level testing for JettySolrRunner.

I was creating a new MiniSolrCluster test because I needed to have the ability to define multiple cores and I was never able to get the test to work via eclipse, traced it down to be this issue.

Sent from my iPhone

Any plans on porting this into the 5x branch? Also, do we know how this will behave with the fancy new startup scripts? It appears clients would need to configure a few things in both the start.in.sh file + the start.ini file as it is sitting right now. Additionally, here are a few potential issues I came across:

1. jetty-ssl.xml
   • Has a bunch of properties that aren't prefixed with 'jetty' ie. 'ssl.port' & 'ssl.timeout' vs 'jetty.ssl.port' & 'jetty.ssl.timeout'.
   • Keystore & Truststore path are always relative to `<Property name="jetty.base" default="." />` which could get annoying for some people if they want to specify an absolute path

In the first comment on SOLR-7339, Shalin has indicated that the upgraded Jetty will be backported to branch_5x after 5.1 is released. The 5.1 release has happened, so expect some action soon. I have no idea what kind of schedule Shalin has for working on this, so I can hope it will be in time for the 5.2 release, but I cannot be sure about it.

Looking into how the default jetty 9.2.10 config does ssl, the config in trunk is quite a bit different. In the stock jetty 9, the Connector for ssl is in jetty-https.xml, not jetty-ssl.xml. It uses "https.port" and "https.timeout" as properties.

Sorry for dropping the ball on this one guys. I am starting now to get this into 5x as soon as possible. I'll need some help with the SSL parts because I am not very familiar with that kind of setup. I'll start by upgrading to the latest release 9.2.10.v20150310.

Looking into how the default jetty 9.2.10 config does ssl, the config in trunk is quite a bit different. In the stock jetty 9, the Connector for ssl is in jetty-https.xml, not jetty-ssl.xml. It uses "https.port" and "https.timeout" as properties.

Yeah, the stock jetty 9.2 comes with both jetty-https.xml and jetty-ssl.xml and https depends on ssl. It wasn't clear to me if both were required or not. Once I upgrade to 9.2.10 then I'll try to make things as close to the stock install as possible.

Patch to upgrade trunk to jetty 9.2.10

Commit 1675261 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1675261 ]

SOLR-4839: Upgrade jetty to 9.2.10.v20150310 in trunk. Also, moved the change log entry to 6.0.0 until we backport to 5x.

Here's patch which conforms our jetty config files to the ones shipped with 9.2.10 more closely.

1. jetty.xml configures the thread pool and common configuration used by both http and https connectors
2. The Connector for ssl is in jetty-https.xml, not jetty-ssl.xml
3. All property names are now prefixed with "solr.jetty." so e.g. https.port becomes solr.jetty.https.port
4. Keystore & Truststore path are no longer relative to jetty.base but they default to ./etc/solr-ssl.keystore.jks as in 5x.

I am inclined to change jetty.port to solr.jetty.port and jetty.host to solr.jetty.host too but that change needs fixes to start scripts and build scripts etc so I will not attempt that in this issue.

This patch fixes the smoke tester to work with the new Jetty configuration. The smoke tester looks for the following string 'Started SocketConnector@0.0.0.0:8983' but the new jetty prints "Started ServerConnector@55b53d44

" to the logs.

At this point, all tests, precommit and smoke tester passes. I'll commit this and then focus on getting SSL support to work.

Commit 1675337 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1675337 ]

SOLR-4839: Make our jetty configs resemble stock Jetty 9.3 configs more closely. Thread pool and common config goes to jetty.xml. All property names are prefixed with solr.jetty. SSL keystore paths are now absolute.

Changes:

1. Threadpool configs: idleTimeout=5000 (5 seconds) and stopTimeout=60000 (1 minute) respectively to match what existed inside JettySolrRunner
2. sendServerVersion is set to false by default to match branch_5x
3. HTTP and HTTPS connector idleTimeout defaults to 50000 ms to match branch_5x
4. HTTPS default port property is called solr.jetty.https.port and defaults to 8983
5. I removed the keyManagerPassword property from jetty-ssl.xml because 1) it was not specified in branch_5x as well and 2) from reading the docs, it defaults to the keyStorePassword but if we ask the user to change it from the solr.in.sh then we must introduce another property which should normally be always set to the same as keyManagerPassword. I don't know if/how this property is useful so I have removed it for the time being. Perhaps someone more knowledgeable about security can chime in here.
6. SSL property names are changed as follows to fix related issue SOLR-7449 so that changes to property values in bin/solr.in.sh is propagated to jetty:

   <Set name="KeyStorePath"><Property name="javax.net.ssl.keyStore" default="./etc/solr-ssl.keystore.jks"/></Set>
     <Set name="KeyStorePassword"><Property name="javax.net.ssl.keyStorePassword" default="secret"/></Set>
     <Set name="TrustStorePath"><Property name="javax.net.ssl.trustStore" default="./etc/solr-ssl.keystore.jks"/></Set>
     <Set name="TrustStorePassword"><Property name="javax.net.ssl.trustStorePassword" default="secret"/></Set>
     <Set name="NeedClientAuth"><Property name="jetty.ssl.clientAuth" default="false"/></Set>
     <Set name="WantClientAuth"><Property name="jetty.ssl.wantClientAuth" default="false"/></Set>
   

7. bin/solr and bin/solr.cmd enable the appropriate module name in jetty (http or https)
8. Removed unused properties in ssl.mod
9. Deleted the start.ini altogether as it was serving no need
10. It is no longer possible to run java -jar start.jar directly because by default neither http nor https module is enabled. The bin/solr script is the only supported way of starting solr. This is required because if we enable the http module by default from inside start.ini then it is not possible to disable http and enable https using command line arguments to start.jar. Instead both are enabled and Solr ends up trying to listen using both http and https connectors.

I tested using the SSL setup documentation in the reference guide and it works with no modifications required to the setup steps.

The tests and precommit passes at this point. I cannot test the smoke tester until I commit so I'll commit this changes and then run the smoke tester.

This patch removes the property SOLR_SSL_PORT in solr.in.sh because there is really no need for another way to configure a port for our start scripts. We can use the regular -p switch or SOLR_PORT property to configure the port in both http and https/ssl modes.

Commit 1675619 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1675619 ]

SOLR-4839: SSL support with Jetty 9. Also fixes SOLR-7449 on trunk.

Looks good, though we might want to think about not reusing the javax.net.ssl.* for the jetty key/trust store configuration. I could think of a few cases where you might want to make the two different, ie one value for the client request and one value for the jetty connector, unless of course the recommendation is to only use self-signed certs for both client and server. Though, maybe the solr.in.sh could have something like:

SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.jks
SOLR_SSL_KEY_STORE_PASSWORD=secret
SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.jks
SOLR_SSL_TRUST_STORE_PASSWORD=secret
#### OVERRIDE PREVIOUSLY DEFINED SSL VALUES FOR HTTP CLIENT IF NECESSARY ######
#SOLR_SSL_CLIENT_KEY_STORE=
#SOLR_SSL_CLIENT_KEY_STORE_PASSWORD=
#SOLR_SSL_CLIENT_TRUST_STORE=
#SOLR_SSL_CLIENT_TRUST_STORE_PASSWORD=

Then the solr startup script can set the javax.net.ssl.* system properties for the client side + create something like jetty.ssl.truststore/keystore/etc on the jetty server side. This would allow a little bit more flexibility for people who might want to use a different certificate or trust store between the http client and server, though this really is getting more on a fringe use case.

Also, if you want to use the javax.net.ssl.* stuff I believe you need to swap

<Property name="javax.net.ssl.keyStore" default="./etc/solr-ssl.keystore.jks"/>

with

<SystemProperty name="javax.net.ssl.keyStore" default="./etc/solr-ssl.keystore.jks"/>

(note the SystemProperty vs Property).

Thanks Steve. I added your suggestions to this patch (SOLR-4839-separate-client-ssl-options.patch) to separate the client and Jetty SSL config.

I don't think replacing Property with SystemProperty is necessary because "Property" can also be set via SystemProperty plus it can also be overridden from one of the module files. However the way we're using them both Property and SystemProperty are equivalent.

I've manually tested these changes on both linux and windows to create single node as well as cloud setup with SSL. All tests and precommit passes. I'll commit this shortly.

This breaks the current set of instructions on how to setup SSL so it will have to be mentioned in the upgrade notes for 5.2 and the reference guide will also need to be changed. I'll do that once I finish the backport to branch_5x.

Commit 1676102 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1676102 ]

SOLR-4839: Separate jetty and client specific SSL properties

Here's a patch to upgrade branch_5x with Jetty 9. All tests and precommit pass.

This also undoes some of the fixes made in SOLR-7429 simply because Jetty 9 has no problem in loading the servlet-api-3.1.0 jar from server/lib

Commit 1676113 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1676113 ]

SOLR-4839: Add upgrade notes, move entry to 5.2.0. Added entry for SOLR-7449

Commit 1676114 from shalin@apache.org in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1676114 ]

SOLR-4839: Upgrade Jetty to 9.2.10.v20150310 and restlet-jee to 2.3.0. Also fixes SOLR-7449. Merges commits r1649552,1649571,1649584,1649689,1650169,1657495,1675261,1675337,1675619,1676102,1676113 from trunk.

Commit 1676116 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1676116 ]

SOLR-4839: Re-organized upgrade notes and added notes about removing the ability to run java -jar start.jar directly and removal of SOLR_SSL_PORT property.

Commit 1676118 from shalin@apache.org in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1676118 ]

SOLR-4839: Fix bad merge which messed up whitespace in smoke tester script

Commit 1676127 from shalin@apache.org in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1676127 ]

SOLR-4839: Re-organized upgrade notes and added notes about removing the ability to run java -jar start.jar directly and removal of SOLR_SSL_PORT property.

Smoke tester and builds have passed on Jenkins. See https://builds.apache.org/view/All/job/Lucene-Solr-SmokeRelease-5.x/260/

I'll close this issue now. Let's create new issues if we need to fix something. Thanks to everyone who contributed!

Commit 1676350 from hossman@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1676350 ]

preemptive cleanup of 'Upgrading' section for 5.2 (SOLR-7325, SOLR-7336, SOLR-4839)

Commit 1676351 from hossman@apache.org in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1676351 ]

preemptive cleanup of 'Upgrading' section for 5.2 (SOLR-7325, SOLR-7336, SOLR-4839 - merge r1676350)

Commit 1676352 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1676352 ]

SOLR-4839: Fix typo

Commit 1676353 from shalin@apache.org in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1676353 ]

SOLR-4839: Fix typo

Commit 1676354 from shalin@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1676354 ]

SOLR-4839: Disable SSLv3 (POODLE) by default from our SSL config. Also added credits for Steve Rowe and Steve Davids.

Commit 1676355 from shalin@apache.org in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1676355 ]

SOLR-4839: Disable SSLv3 (POODLE) by default from our SSL config. Also added credits for Steve Rowe and Steve Davids.

Thanks to Steve Rowe for reminding me to disable SSLv3 from our default SSL config.

Commit 1676508 from shalin@apache.org in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1676508 ]

SOLR-4839: Remove servlet-api as an explicit dependency from solr/core to match trunk

jetty-distribution-9.2.11.v20150529.zip ?

9.2.11 is released.

Bill, Solr 5.2 release process is already underway with 9.2.10.v20150310 and this issue has been resolved. Please open a new issue for upgrading to 9.2.11 so that we can take it up for 5.3.

Bulk close for 5.2.0.