@Ryan Mckinley: Many thanks for attaching the patch here. It is most useful.
@Hoss Man: Consider this usecase.Take your favorite ecommerce site ( say newegg.com, ebay.com etc ). Notice that they have some kind of category hierarchy. Each category has category attributes ( say Brand ) with category sensitive possible values(Apple/Samsung for cell phone and Sharp/Samsung for HDTVs) (. In these cases the number of categories specific attributes are in 10's of thousand. It is not realistically possible to create such a schema so that every category specific attribute is mapped to a solr field. However, you can store the category specific attributes per category as a json string.
Now, you do need to filter by category specific attributes. Say you are searching for HDTVs and you only want to see those manufactured by Samsung. As is, solr will not allow you to search in a field which looks like this:
something like fq=categoryattribute:"name":"brand","value":"samsung" ( properly escaped ) doesn't work
Enter the awesome jsontokenizer written by Ryan McKinley. This allows the same field to be indexed as json and
something like fq=categoryattribute:"name:brand" AND categoryattribute:"value:Samsung" works.
Happy to provide more information if needed. Also happy to take the slap if I'm missing something obvious here.