Description
Add a Velocity - http://velocity.apache.org - response writer, making it possible to generate a decent search UI straight from Solr itself. Designed to work standalone or in conjunction with the JSON response writer (or SolrJS) for Ajaxy things.
Attachments
Attachments
- SOLR-620.patch
- 4 kB
- Matthias Epheser
- SOLR-620.patch
- 9 kB
- Erik Hatcher
- SOLR-620.patch
- 10 kB
- Matthias Epheser
- SOLR-620.patch
- 3 kB
- Erik Hatcher
- SOLR-620.patch
- 3 kB
- Erik Hatcher
- SOLR-620.zip
- 1.13 MB
- Erik Hatcher
- SOLR-620.zip
- 1.07 MB
- Erik Hatcher
- SOLR-620-velocity.properties.patch
- 4 kB
- Koji Sekiguchi
- SOLR-620-velocity.properties.patch
- 2 kB
- Koji Sekiguchi
Issue Links
Activity
The JARs used to build this example (probably not the most current versions, but what I happened to have laying around).
There are a zillion bells and whistles that can be added to this, lots of Velocity parameters that can be controlled. We'll probably want to have some custom Solr macros to make templating life a lot easier. For example, we'll need to make navigating the DocList easier.
Just added also the request object to the context for my very first implementation of a solrjs, so the searcher is avaliable for $request.getSearcher().doc(id).
I think there are two possible ways to get make easier:
- Wrapping the resonse in an object (maybe SolrReponse from solrj), providing handy getters and loops etc.
- Creating velocity macros
I'll experiment a bit with both solutions and comment again.
perhaps look at convrting the DocList to a SolrDocumentList. Then velocity could use the solrj functions.
take a look at: org.apache.solr.update.DocumentBuilder – it has a function loadStoredFields that you can use to convert a Document to a SolrDocument.
Here's an updated patch with a bit more meat to the default.vm, also adding in the SolrQueryRequest to the Velocity context.
I've done it the hard way for now, by navigating the API as rawly as it gets to display the documents. Doing it this way for a bit will inform what kind of macros or other context helpers we need to make the view as clean as possible.
Matthias - I removed your patch to keep it clean here. If you name the patch SOLR-620.patch, it will grey out previous versions. Two comments on your patch, be sure to flag your patch as being ok to use within the ASF, and also be sure to svn add (locally only of course) the .vm files so they appear in the patch too.
Another handy method to get all the documents is SolrIndexSearcher#readDocs(Document[] docs, DocList ids) - however it is probably best for memory usage (or is it?) to continue to use the DocList#iterator() so only a single document is loaded at a time in case there are large field values or a lot of documents requested. Although this point will be a bit moot when Velocity layouts are implemented (something I plan to do eventually) where the original template is generated into a string buffer and then another wrapping template is generated around it.
however it is probably best for memory usage (or is it?) to continue to use the DocList#iterator() so only a single document is loaded at a time
Yes. That's why I deferred loading documents until response writing.... so things could be streamed one doc at a time.
I've done it the hard way for now, by navigating the API as rawly as it gets to display the documents. Doing it this way for a bit will inform what kind of macros or other context helpers we need to make the view as clean as possible.
Played around a bit with macros and their capabilities, I think creating a java context helper class is the better way. The velocity template should make it easy to create very custom html. With macros, we could "hide" the complex method calls, but eg. creating a shortcut for a loop AND customize the html inside this loop will not be possible easily.
And, correct me if I'm wrong, your "shortcut setters" like "#set($iterator=$response.values.response.iterator())" have to be declared in every template file, cause template libraries cannot share variables, and macros only produce Strings and no other return types.
So I suggest a small java wrapper object that still uses the doclist for performance considerations and only provides handy convenience methods like
getResultIterator() -> iterator of ids, used in #foreachs
getDocFieldValue(docId, fieldName)
etc.
so a custom result view looks like this:
#foreach($docid in $helper.getResultIterator())
<li>$helper.getDocFieldValue(docid, "title")</li>
#end
This object can be passed to the velocity context and may be accessible to all custom templates, without changing the data's nature (doclist to momory objects or the like). I'll have a test version of thos approch ready this evening.
Isn't this a good candidate for a contrib project?
- It introduces quite a few dependencies Solr core does not need
- Most of the users do not deploy the webapp in Solr itself (for security reasons)
Sure, contrib is fine. Where it ends up isn't a big deal to me.
As for how Solr is deployed - having the web tier and Solr be one and the same (in a read-only configuration of course) is a pretty compelling way to deploy, I'd think. Why separate Solr and the UI when using load balanced replicated servers? Of course it depends on the needs of the application, but keeping a separate web app to regurgitate what Solr returns can be an unnecessary deployment hassle.
Looking at a recent version of this, it seems best if we can reuse the NamedList<Object> parsing from solrj rather then have the velocity writer add helper functions to access parts of the response.
I just posted SOLR-641 to expose this parsing form EmbeddedSolrServer
check SOLR-654 for a way to construct a SolrResponse from a given class – this way the velocity writer can have access to any utility functions a SolrResponse implementation has.
According to the patches Ryan mentioned above (SOLR-641 and SOLR-654, I created an update of VelocityResponseWriter.
It now contains the following configuration possibilities:
- vl.template: The template file to use inside conf/velocity/
- vl.json: If this param is true, the output of the template is wrapped inside a javascript function, eg myFunction( <actual output> ). This is necessary if we want to use JSON requests out of a javascript client.
- vl.content: The content type of the response. Default is text/html in standard mode, and text/json in vl.json=true mode.
- vl.response: To provide an implementation of SolrResponse inside the template, specify the class name of the implementation. For convenience, it looks inside the package "org.apache.solr.client.solrj.response", so you only have to choose eg. QueryResponse, LukeResponse, MultiCoreResponse. Custom classes may be accessed using the full qualified class name, eg. my.custom.package.CustomResponse
Regardless of the vl.response, there is always an instance of RawRequestHelper inside the context, this provides access to the SolrQueryRequest and SolrQueryResponse, as well as convenience methods like getResultIterator() or getRequestParam()
Matthias - I'm perplexed by the JSON feature. The intention was to use the Velocity writer to produce templated output, but if you want JSON data, use wt=json, not the velocity one.
the JSON feature gets around a javascript limitation for cross-site scripting. If you enclose your HTML within a JSON tag, you can access from any server – otherwise it has to come from the same machine.
the json feature just sends your velocity template output (perhaps html) surrounded with:
{ result: "your template output here (escaped for json)" }
Would it make sense to allow passing of the velocity template in a parameter for extra on-the-fly flexibility?
That's an interesting idea, Yonik. Easy enough to implement. I'm not quite sure how useful it'd be - I'm curious about what use cases this would be leveraged. I'll toss it in though.
The same feature would be useful to the XsltResponseWriter also, I'd think.
Attached is a new patch. Sorry Matthias, this doesn't included your changes (mostly the JSON wrapper, something I've yet to fully understand the need for).
This new patch allows for templates to be specified as parameters. &template.hit=$doc.get('id'), for example, overrides the hit.vm template found in conf/velocity.
http://wiki.apache.org/solr/VelocityResponseWriter contains usage instructions and examples.
This .zip supercedes the previous patches (except for the JSON wrapping feature from Matthias - no worries, that part will come back as needed). The .zip slides into the contrib/ structure as "velocity". Unzip, then run "ant dist" in contrib/velocity to have the example configuration all set up for this.
Updated .zip. Simply extract to contrib, then start up the example (from example/ subdirectory): java -Dsolr.solr.home=../contrib/velocity/src/main/solr/ -jar start.jar
This adds pagination, CSS, a decent facet navigation, reasonable document view, and JQuery integration (just for toggling explanation hit detail for now).
Looks good.
I think you should go ahead and commit it. It is easier to keep track of the changes once it is committed.
Thanks Ryan! Committed. Needs work, but it's a good starting point, and usable as is.
don't know why, but FF 3.0.3 (on mac) does not display anything unless we change:
Index: contrib/velocity/src/main/solr/conf/velocity/browse.vm =================================================================== --- contrib/velocity/src/main/solr/conf/velocity/browse.vm (revision 709500) +++ contrib/velocity/src/main/solr/conf/velocity/browse.vm (working copy) @@ -7,7 +7,7 @@ <head> <title>#param('title')</title> <meta http-equiv="content-type" content="text/html; charset=UTF-8"/> - <script type="text/javascript" src="/solr/admin/jquery-1.2.3.min.js"/> + <script type="text/javascript" src="/solr/admin/jquery-1.2.3.min.js"></script> <link rel="stylesheet" type="text/css" href="/solr/admin/file?file=/velocity/main.css&contentType=text/css"/> </head> <body>
What are your thoughts on what should go in the velocity contrib src/main/solr files?
I don't like that it duplicates all the stuff in /trunk/example/solr
In my view (influenced by wicket), it would be great if the velocity templates were packaged in the .jar files rather then in the config directory.
As for the duplication - I agree.... really all I'm after is getting an example configuration, much like data import handler (which also duplicates an entire configuration, mostly). All this VelocityResponseWriter needs is to be registered in solrconfig.xml (where's Spring when you need it, such that dropping in a JAR is all that is needed to wire in a new component?! Suggestions on improving, or just hardwire it into the default example config, are welcome?
Regarding Velocity template location, Velocity has a very flexible resource loader mechanism. Right now it's hardcoded to use the custom parameter loader (&template.<template name>=<body>) or the file system. Just wiring in the built-in classpath resource loader won't be enough... we need a SolrResourceLoader pass-through to pick up templates in JAR files in conf/lib - that'll be real handy.
I'll work on the JAR loading capability, shouldn't be too hard to toss in there.
Ok, got your JAR loader in place. At this point, templates get found first in params (&template.name=body), then the file system (either v.base_dir property value, bad name I know, or conf/velocity), then the JARs. This gives overridability down to the request level.
The one caveat to the JAR loader... it's actually just a front-end to SolrResourceLoader#openResource(), so it can open files in the conf directory (or via absolute path, it seems!), as well as from conf/lib/*.jar's and the Solr classpath itself.
We'll make the Velocity properties more flexibly managed as we evolve this.
Templates on the file system really make the most sense to me, but we'll let Velocity's flexibility shine through.
Just noticed that this contrib is not added into the release/signing/packaging stuff in the top level build.
As discussed on the mailinglist, this patch adds the removed json wrap and response writer capabilities.
I have added in the json escaping/wrapping feature, renamed all parameters (including template overrides) to use the "v." prefix. But I did not add the rawResponse feature yet. It needs more thought, and no known example is using it yet. I'd prefer that SolrJ's SolrQuery and SolrResponse objects be put into the Velocity context - making this portable between embedded and HTTP Solr calls. Using the SolrJ API will alleviate the need that rawResponse was filling, I believe.
I got my HTML page with garbage characters because my template encoding is UTF-8 but velocity expects it ISO-8859-1 as default (default velocity.properties file). I think it is necessary we can override default properties. How do people think applying the patch?
Yeah, I figured we'd need to provide a mechanism to load velocity.properties, or at least specify them in some manner through config. Does the engine.init calls after setting some properties lose those values originally set? Maybe velocity.properties should be loaded from SolrResourceLoader so that it can live on the classpath?
Does the engine.init calls after setting some properties lose those values originally set?
According to the Velocity document, the sequence of ve initialization will be engine.setProperty(ies) then engine.init.
http://velocity.apache.org/engine/releases/velocity-1.6.2/developer-guide.html#separate
And I checked VelocityEngine through debugger. It was no problem.
Maybe velocity.properties should be loaded from SolrResourceLoader so that it can live on the classpath?
Right. I updated the patch.
I'd like to commit if there is no objections.
I think we can resolve this issue and open new ones for any additional work needed on the VelocityResponseWriter.
First draft - very basic operation. Must also add Velocity, Commons Lang, and Commons Collections to the classpath.
Example: http://localhost:8983/solr/select?q=ipod&wt=velocity
With an optional template parameter, specifying the template name under conf/velocity (&template=default)