Starting implementation, setting the "shardMapping" parameter to any value in the request will add the field "shard" to each response document containing the shard as specified in the request.

• Currently only implemented for XML responses.
• No tests.
• When a document is found in multiple shards, the last one sets the value, the others are lost. Returning an array of all the shards would probably better.

This code is almost untested.

I don't think we need to worry about returning an array of all shards... they are supposed to be disjoint, but we handle gracefully if an id is repeated (instead of blowing up).

Instead of "shard", maybe we want to pick something that won't clash with a real field name quite so easily?
_shard? (yes it's uglier).

What about putting the docid->shard mapping elsewhere in the response rather than actually on the document? Like explain and highlight info, for example.

What about putting the docid->shard mapping elsewhere in the response rather than actually on the document?

I've never been sure what the right answer is here. Putting it in a different place sometimes seems cleaner, but sometimes seems like it just makes responses harder to read, and forces users to do their own id based correlation.

I've also thought about a "meta" part to a document that contains other information specific to the document besides stored fields.

Well, can you filter/query/sort by the contents of this "shard" field? If not, it doesn't belong in the doc block, IMO

When you add new fabricated fields to the document return, please use a non-standard naming convention, like "&shard" and "&score". Adding simple alpha words as fields will clash with someone's schema.

Well, can you filter/query/sort by the contents of this "shard" field? If not, it doesn't belong in the doc block, IMO

You can do any of those with stored (non-indexed) fields either.

I don't think we need to worry about returning an array of all shards... they are supposed to be disjoint

I think as the idea behind the issue is to use this information to programmatically update/delete documents, we should return an array of shards. Consider the scenario where 2 sets of shards index the same information for redundancy purposes. For normal queries, you would send requests to one set of shards. If you want to delete a document, it would be nice to be able to send one request to both sets of shards and get all the required information with a single request instead of having to query each set individually.

What about putting the docid->shard mapping elsewhere in the response rather than actually on the document?

That assumes that there's a unique key which we can use to link the two pieces of information. That's probably a reasonable assumption, but in my opinion we shouldn't impose this restriction unless it's really necessary.

I've also thought about a "meta" part to a document that contains other information specific to the document besides stored fields.

Ah, I do like that idea.

Well, can you filter/query/sort by the contents of this "shard" field? If not, it doesn't belong in the doc block, IMO

It's the same thing for the score field though.

Attaching new patch which, inspired by Yonik's suggestion, returns something like

  <doc>
  <lst name="meta">
        <str name="shard">localhost:8983/solr</str>
  </lst>
  <str name="id">MA147LL/A</str>
  <str name="name">Apple 60 GB iPod with Video Playback Black</str>
 </doc>

The parameter format changed as well, to get the doccument -> shard mapping, add &metadata=shard to the request. The patch adds a new field for metadata to SolrDocument. This should probably also be used for score and split out into a separate issue if that general direction is ok.

Issues with the current implementation:

• Only works with XML responses and lists of SolrDocuments.
• The document found in multiple shards issue needs to be clarified.
• Tests.

I would be keen to test this using Ruby - can you add this for the Ruby Request as well as XML in patch?

Attaching new patch that actually applies again (sorry, this issue had escaped my attention before) and adds metadata output for Ruby and JSON.

Syncing patch with trunk.

The latest patch doesn't seem to compile anymore, I get:

{{

compile-solrj:
[javac] Compiling 1 source file to /Users/bwhitman/outside/solr-trunk/build/solrj
[javac] /Users/bwhitman/outside/solr-trunk/src/common/org/apache/solr/common/SolrDocument.java:50: cannot assign a value to final variable _fields
[javac] _fields = new LinkedHashMap<String,Object>();
[javac] ^

}}

Attaching new patch which compiles again.

Sooooo....what do people think of "meta"?
It seems like it could make things easier on clients - removing the need for correlation-by-id.

Sooooo....what do people think of "meta"? It seems like it could make things easier on clients - removing the need for correlation-by-id.

I'm +1 on the idea. Let's choose a better name than 'meta' with less likelihood of collision, say 'meta'?

I like the concept – it could help for many things, and make some client code a bit easier.

The simple use case is to augment each document with fields that do not exist in the schema. This could be things like the shard, the score, a calculation (perhaps the output of a function query)

This may also be someplace to consider putting other information that currently requires correlation-by-id. Consider highlighting, MLT, some debug info...

Here is an updated patch that:
1. Calculates the Set of returned fields once per requres (rather then for each document)
1. Attaches the meta return field list to the SolrQueryResponse
1. implements a ReturnFieldList that will match "*" to return any meta field (NOTE, we should consider http://wiki.apache.org/solr/FieldAliasesAndGlobsInParams)

I like that this gives us an extendible place to augment each document without having to map the ID.

I don't like the potential name conflict using "meta" (or any string) in the field list. Since you have to explicitly turn on the query, i guess it is ok; BUT it makes things difficult for parsers. Is the "meta" field the special key or a field?

Since solr does not support NamedList as a field, we could asume that any <lst is the special meta field, but that seems kinda ugly.

i haven't looked at any of the patches here, but the simplest way to avoid field name collision problems would be to make the client specify the name of the "meta" field when asking for it

examples...

binary might decide to leave it out completely

The behavior should be consistent irrespective of the output format we choose. The binary format should have no problem in handling this usecase the same way xml format handles it

i wasn't suggesting that the binary format couldn't handle it ... i was just suggesting that if the client fails to specify a "meta" fieldname for the response writer to use when including the metadata in each doc, the behavior can be implementation (ie: response writer) dependent.

i'm speaking to what the "spec" for metadata fields could look like, not what the implementations should look like.

I'll take this one on for 1.4...

I incorporate hoss' suggestion and then we can see how we like it.

Ryan: might be worth while to split the jira issue ... create a new issue for an internal API to add per doc metadata and use of this metadata in at least 2 response writers; then make SOLR-705 (shard mapping) and SOLR-1298 (function query results) dependent on the new issue and sanity test of the new internal APIs.

I go back and forth on the "meta" thing...

On one hand, if one is looking at the output, it makes perfect sense to have a separate meta section per document.
However, when one looks at it from a client API perspective (how one asks for the value of a particular metadata value) having two different ways to access values ("real" fields vs "meta" fields) doesn't seem desirable.

From a client coding perspective, consistency is nice:
sdoc.get("id")
sdoc.get("_shard")

After all, many of the stored fields of a document are actually just metadata too. So an alternative is simple convention... metadata fields start with an underscore, and no more work needs ot be done at the client side.

But I'm really not convinced either way

If we do go with "meta", I'm also not concerned with the hypothetical field-name collision... this is a one time thing, and hard-coding it to "meta" makes things simpler and more predictable.

Let us have a special field of something like "meta" (to minimize conflict) which can be used for any future metadata.so, the client code would look like

sdoc.get("id")
NamedList meta = sdoc.get("_meta_");
meta.get("shard");

On one hand, if one is looking at the output, it makes perfect sense to have a separate meta section per document. However, when one looks at it from a client API perspective (how one asks for the value of a particular metadata value) having two different ways to access values ("real" fields vs "meta" fields) doesn't seem desirable.

I'm trying to look at it from an internal datastructure perspective, and a client code perspective. From an internals perspective, keeping the data isolcated makes sense – one set comes straight from the Documents in the index, the other is computed – so they should be seperate datastructures internally in solr, one hanging off the other.

Then the response writer can decide how it wants to deal with those data – for response writers where nested data structures are easy (most of them) this data can be represented cleaning ... or we could add options to flatten the data (using some prefix type option to say all metadata data properties should become fields with the same name prefixed by "_") so that the client can't tell the difference between them ... if we make the internal representation a flattened list of pairs, then we tie the hands of hte response writter.

Ditto for the client library – the more structure we maintain as the data makes it's way to the client, the more options we have as to if/when we flatten it. preserving structure allows code to flatten at anytime if it wants to, so let's go with the option that has the most flexibility and get the best of both worlds.

Moving this issue to 1.5 so that the details could be worked out with less haste.

http://www.lucidimagination.com/search/document/1f2e56f58162679d/response_writers_and_doclists

Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...

http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E

Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.

A unique token for finding these 240 issues in the future: hossversioncleanup20100527

SOLR-1566 is now commited, so this should be pretty easy to get wrapped up

I don't have any experience with sharding so someone else will need to step up for this one.

Looking at it quickly, if the process that fills in the DocList knows what shard it is, then it would be trivial to add something similar to the value augmenter. If the SearchComponent needs to set the field it will be a little more complicated

Bulk move 3.2 -> 3.3

added &shard.url=xxx to distributed requests and return that with a DocTransformer

Something to keep in mind is the difference between logical shard and physical shard replica.
i.e. "shard1" can be located at localhost:8983/solr/shard1 and localhost:7574/solr/shard1

Both pieces of info can be useful.

So with ryan's last commit, [shard] gets you something like localhost:8983/solr/shard1
We don't have to implement returning the other part now... but we should think about the naming.

In the code, I sometimes used "slice" to mean logical shard (a logical slice of the complete index), to avoid overloading "shard"... but I'm not sure that won't cause more confusion than it's worth. So for a future logical shard name, perhaps [shard_id] ?

interesting. What defines a 'slice'? Could it be a system property, or something in SolrConfig?

If I understand what you are saying, it would be a label on cores that are logically identical.

interesting. What defines a 'slice'? Could it be a system property, or something in SolrConfig?

It's well defined within SolrCloud... you can actually add &shards=shard1,shard2 and solr will do the mapping from those logical shards to physical shards (via cluster state in zookeeper) and do a load-balanced request across them.

OK, that suggests that the parameter should be 'shard.id' rather then 'shard.url' – since in SolrCloud it is not a url. Maybe we should also send shard.url so that we do know the URL even within SolrCloud. Then we should add another transformer for [shard_url]