I think they should be inline, as they are just values associated with a document. I think putting it in some other list is sticking too literally to what Lucene calls a field, which I don't think Solr has to do that. One could easily imagine a Solr component that brought in a database or other storage repository for supplementary fields and it should all be seamless to the client.
I definitely agree that one shouldn't see a field in Solr as a field in Lucene. That said, I think do have a tendency to see a field in Solr as somehow bound to the Solr schema.
One thing to notice is that eventually we end up with the same discussion regarding this feature in the context of different issues, let it be highlighting or field collapsing. In some cases it feel just "right" to return the data as a field in a document, in other places it feels "right" to have as something else. It is true that when you interact with solr directly (specially if you do it manually) you certainly know what queries you send, what functions you request and what you should expect in the result. But from experience, a lot of times you try to automate things a bit and creating a well structured and descriptive protocols is the safe way to enable that.
I don't want to have to go look it up in some other list while I am iterating over my results when all the other values I'm displaying/using are right there associated with the document.
Having a sub-section under each documents still associates it with the document. The way I see it, It's like OOP... you can have a Person class that holds all the information of the person it it as primitive fields, or you can group related data, like address info, int a separate Address class.
That being said, it could be useful to add an attribute that indicates it is a generated name
That's one way to group fields together, but if you're already doing that, then why not go all the way? If you need to distinguish between generated and non-generated names, why not make it simpler and just separate the two in a different list? (To continue the analogies line I started above ) it's like XML, you can have a single level hierarchy were each element defines attributes to relate it to other elements, but a more suitable solution would just be to group all related elements under one parent element.
I'd even argue that highlighter results should be inline, too, but that is a different issue and a bigger can of worms since it has a well used API already.
In some cases it might be (well it just is) more appropriate to have the highlighting inlined. In other cases it might not be possible, specially with some of the latest requests to have highlighting functionality available for arbitrary text loaded from anywhere (which I believe will lead for a highlighting component/requestHandler that will be independent of the query component).
Not saying this is right or wrong, but I think it would be useful to document here the rationale about why not to do it. Is it just b/c that method is expected to do, more or less, what the Lucene IndexSearcher does?
I guess so... I guess SolrIndexSearcher is in fact a Lucene IndexSearcher which is the source for this association. In some ways I think it's also relates a bit to the response structure (not directly though, but conceptually)... if the IndexSearcher represents Lucene and the document contains fields coming from other sources as well, perhaps this functionality of gathering all these fields (/metadata ) should be done in a higher level where SolrIndexSearcher just serves as on "field source". The main reason why Chris's patch puts this functionality in the doc() method of the SolrIndexSearcher is simply because it's the easiest and the simplest solution right now... and I don't thing there's nothing wrong with that... simple is good! Even with this solution as it is, the "field sources" are still abstracted away in the form of a "FieldValues" or "DocumentMutator", so architecture-wise I don't see leaving it as is will compromise anything.