Patch implementing the idea.

No (functioning) tests yet.

I've started the example server using the default solr home and the multicore solr home, and requests to all methods are functional from curl.

The patch adds two dependencies: Restlet and the Restlet servlet extension. All REST methods are implemented as Restlet ServerResource subclasses, which delegate to new self-reporting methods on IndexField and FieldType, the implementation of which was inspired by/stolen from LukeRequestHandler.

SolrDispatchFilter figures out the core, creates a SolrRequest and a SolrResponse, sets them on SolrRequestInfo's thread local, then passes the request (via filter chaining or request forwarding) to the Restlet servlet defined to handle schema requests. Based on the URL path, the Restlet servlet's router then sends the request to the appropriate ServerResource subclass, where the response is filled in.

There is no RequestHandler involved in servicing these requests.

I've turned off Restlet's content negotiation facilities in favor of using Solr's wt parameter to specify the ResponseWriter.

At present, both GET and HEAD requests work for all six requests. (Restlet uses GET methods to service HEAD requests, so there was very little coding required to do this.)

I'm interested in what other people think of using Restlet in Solr - this issue, in part, is about exploring how to do that.

Restlet brings some baggage:

• Non-RequestHandler-based Restlet actions aren't available (as currently written anyway) via EmbeddedSolrServer, which only knows how to deal with requests that have RequestHandlers.
• Restlet's artifacts aren't deployed to Maven Central - instead, they host their own Maven repository. I was worried that having dependencies drawn from 3rd party Maven repositories would cause trouble, so I deployed to the ASF staging repository a fake Solr release including the two Restlet dependencies in the Solr core POM, and the quality checks performed as part of "closing" didn't flag this as a problem, so I think using Restlet will not block Lucene or Solr from deploying to Maven Central.

Restlet should make some things easier, though, e.g. the PUT and DELETE methods are usable.

I have not looked at the patch yet, but in general, I like restlet. It's dual licensed right? One of the licenses compat with Apache?

I have not looked at the patch yet, but in general, I like restlet. It's dual licensed right? One of the licenses compat with Apache?

It's quintuply licensed: Apache 2.0 / LGPL 3.0 / LGPL 2.1 / CDDL 1.0 / EPL 1.0

Oh nice, Apache 2 - pretty sure they did not offer that back when I was using it.

No, in depth review, but just took a few minutes to skim over the patch - from a high level, looking good!

I think the restlet integration is fine.

I'd love to redo the coreadmin/collections apis to be a bit more restful with restlet - not sure I want to deal with the back compat ramifications though.

No, in depth review, but just took a few minutes to skim over the patch - from a high level, looking good!

Thanks for taking a look.

I think the restlet integration is fine.

Cool! Since I want to use Restlet for other aspects of SOLR-3251, I thought it would be good to get it in sooner rather than later.

Darn... patch is already out of date.
Do you have some examples of what responses look like?

Darn... patch is already out of date.

I'll put up a more modern version, with most tests in place, in a little bit.

Do you have some examples of what responses look like?

Yes, I do, attaching JSON responses for all six requests here. I haven't investigated yet, but I believe there is an issue with copy fields in both the /schema/fields/ and /schema/dynamicfields/ responses, probably related to SOLR-3798, since I copied the functionality from LukeRequestHandler. But otherwise, I think the response formats are stable now.

Here's my current state.

I refactored REST-friendly stuff out of TestHarness into BaseHarness, which is now extended by TestHarness and a new RestTestHarness. RestTestBase extends SolrJettyTestBase to provide HTTP-based (no SolrServer) test sugar: assertU, assertQ, assertJQ, etc.

I've also added the ability to JettySolrRunner to add extra passed-in servlets, and I use this functionality to add the Restlet servlet that services the six schema resource requests.

There is a test suite for each of the six resources, configured over schema15.xml and solrconfig.xml from solr/src/test-files/.

'ant test' and 'ant precommit' pass for me.

Left to do:

• Add checks for copy fields
• Add checks for the includeDynamic=true functionality for the /schema/fields/ resources - this triggers a search for fields in the index that match dynamic field patterns and are not explicitly defined in the schema. If "fl" fields are specified in the request, only those fields are sought in the index.
• Add Maven dependencies for Restlet and the Restlet servlet extension

One question: both /schema/fields and /schema/dynamicfields accept a "fl" query parameter to restrict the response to the given field(s)/pattern(s), but I didn't do the same for /schema/fieldtypes/, since that didn't seem to fit quite right. Should I add this capability there, maybe with some other query parameter?

Looking good Steve!

It seems like there are multiple use-cases for the fields API...

• One just wants to get all the info/properties for a field, and they don't care how that field is defined.
  For example, they may want to look up field X to see if it's indexed, w/o having to worry about if it's a "dynamic" field or not.
• One wants to get the actual definition definition in the schema (or as close to that as possible), and they would expect to see something very close to what they defined (or sent in using the fields API to create a new field). Showing all possible field properties like "storeOffsetsWithPositions" are going to be potentially confusing since people won't know what they mean and when/why they have to specify them when creating new fields.

It may be possible to satisfy everything with the same API by:

• only returning non-default properties... so if you create an integer field with "indexed=true", that's pretty much all you get back.
• having a parameter to allow requesting all "look through" properties.
• allowing the fields API to be used for dynamic fields also, but provide an indicator about what pattern matched

Also related: SOLR-4210
We should aim for being able to hit any node in the cluster w/o worrying about which nodes are hosting which collections.

We should probably also default to indented JSON output.

Thanks for the review, Yonik.

It seems like there are multiple use-cases for the fields API...

One just wants to get all the info/properties for a field, and they don't care how that field is defined.
For example, they may want to look up field X to see if it's indexed, w/o having to worry about if it's a "dynamic" field or not.

I originally had code to do index lookup, but this API is about the schema, so I took it out, thinking that index queries didn't belong.

allowing the fields API to be used for dynamic fields also, but provide an indicator about what pattern matched

In my current state I do have an "includeDynamic" query param for the /schema/fields/fieldname request, to report the matching dynamic field properties if fieldname isn't an explicitly declared field.

One wants to get the actual definition definition in the schema (or as close to that as possible), and they would expect to see something very close to what they defined (or sent in using the fields API to create a new field). Showing all possible field properties like "storeOffsetsWithPositions" are going to be potentially confusing since people won't know what they mean and when/why they have to specify them when creating new fields.

only returning non-default properties... so if you create an integer field with "indexed=true", that's pretty much all you get back.

having a parameter to allow requesting all "look through" properties.

Good idea, I'll do this.

Also related: SOLR-4210
We should aim for being able to hit any node in the cluster w/o worrying about which nodes are hosting which collections.

Skimming Mark's patch modifying SolrDispatchFilter, I think the schema REST requests will already be proxied, but I'll test to be sure.

We should probably also default to indented JSON output.

OK, I'll switch to that.

Patch, I think it's ready to go. I've also attached updated example outputs.

I've split off a new /schema/copyfields/ request, because it wasn't possible to attach copyFields with subset pattern dynamic field references off of anything in the /schema/fields/ structure. /schema/copyfields/ also now contains maxChars, sourceDynamicBase and destDynamicBase, if applicable. I had to refactor dynamic field handling in IndexSchema.java in order to fix a bug identified in SOLR-3798 - this refactoring+fix is included in this patch.

By default, all requests now exclude default properties. A "showDefaults" query parameter causes them to be included in the response.

By default, all requests are indented JSON.

There are tests for everything, 'ant test' passes under Solr, 'ant precommit' passes, and I've added CHANGES.txt entries.

Also related: SOLR-4210
We should aim for being able to hit any node in the cluster w/o worrying about which nodes are hosting which collections.

I set up a cluster with two collections with different configurations so I could easily verify I was getting responses from the right collection. Turns out that the schema info GET requests were being converted into POST requests as a side-effect of calling con.getOutputStream() in SolrDispatchFilter.remoteQuery(), and failing as a result. In the patch snippet below, which is included in the attached patch, I skip forwarding the request body unless the original request's method is POST:

@@ -353,13 +365,17 @@
       try {
         con.connect();
 
-        InputStream is = req.getInputStream();
-        OutputStream os = con.getOutputStream();
-        try {
-          IOUtils.copyLarge(is, os);
-        } finally {
-          IOUtils.closeQuietly(os);
-          IOUtils.closeQuietly(is);  // TODO: I thought we weren't supposed to explicitly close servlet streams
+        InputStream is;
+        OutputStream os;
+        if ("POST".equals(req.getMethod())) {
+          is = req.getInputStream();
+          os = con.getOutputStream(); // side effect: method is switched to POST
+          try {
+            IOUtils.copyLarge(is, os);
+          } finally {
+            IOUtils.closeQuietly(os);
+            IOUtils.closeQuietly(is);  // TODO: I thought we weren't supposed to explicitly close servlet streams
+          }
         }
         
         resp.setStatus(con.getResponseCode());

The patch also contains some fixed tests for the schema REST API (I didn't update the tests after I removed printing of copyField maxChars when it's zero).

'ant test' under Solr passes, as does 'ant precommit'.

If there are no objections, I'll commit this form of the patch in a day or so.

Steven, that may be a dumb question, but all your sample files have Content-Type: text/plain; charset=UTF-8, is that only for demonstration? because if we follow the RFC4627 it'd be: application/json

Regarding JSON content type, this may be related to SOLR-1123 which makes the content-type configurable, but defaults to something that will display in a browser by default.

Awesome stuff Steve!

(edit fixed location of {{ solrconfig.xml}})

Steven, that may be a dumb question, but all your sample files have Content-Type: text/plain; charset=UTF-8, is that only for demonstration? because if we follow the RFC4627 it'd be: application/json

Regarding JSON content type, this may be related to SOLR-1123 which makes the content-type configurable, but defaults to something that will display in a browser by default.

Yes, by default, the JSON response writer is used, and its configuration in solr/example/solr/collection1/conf/solrconfig.xml (which I used to generate all the example responses attached to this issue) is:

<queryResponseWriter name="json" class="solr.JSONResponseWriter">
   <!-- For the purposes of the tutorial, JSON responses are written as
    plain text so that they are easy to read in *any* browser.
    If you expect a MIME type of "application/json" just remove this override.
   -->
  <str name="content-type">text/plain; charset=UTF-8</str>
</queryResponseWriter>

Hmm, I just noticed that reported dynamic field properties include "dynamicBase", which is stupid:

curl 'http://localhost:8983/solr/schema/dynamicfields/*_i'
{
  "responseHeader":{
    "status":0,
    "QTime":1},
  "dynamicfield":{
    "name":"*_i",
    "type":"int",
    "indexed":true,
    "stored":true,
    "dynamicBase":"*_i"}}

This patch fixes the problem by excluding "dynamicBase" when it's the same as the field "name".

After the latest patch fixing removing inappropriate "dynamicBase" properties, attaching fixed example dynamic field responses.

[trunk commit] Steven Rowe
http://svn.apache.org/viewvc?view=revision&revision=1453161

SOLR-4503: Add REST API methods to get schema information: fields, dynamicFields, fieldTypes, and copyFields. Restlet 2.1.1 is integrated and is used to service these requests.
Also fixes bugs in dynamic copyField logic described in SOLR-3798.
Also fixes a bug with proxied SolrCloud requests (SOLR-4210) when using the GET method.

Committed:

• trunk r1453161
• branch_4x r1453162

[branch_4x commit] Steven Rowe
http://svn.apache.org/viewvc?view=revision&revision=1453162

SOLR-4503: Add REST API methods to get schema information: fields, dynamicFields, fieldTypes, and copyFields. Restlet 2.1.1 is integrated and is used to service these requests.
Also fixes bugs in dynamic copyField logic described in SOLR-3798.
Also fixes a bug with proxied SolrCloud requests (SOLR-4210) when using the GET method.
(merged trunk r1453161)

Closed after release.