Here is my first attempt at a patch that is not currently working. For some reason only the prepare and process methods are being called. It seems that the shards parameter is not being honored like it is in the other distributed components because rb.shards is always null. I have looked at the other distributed components and did not notice them doing anything special with the shards parameter. I have based this code on the information from http://wiki.apache.org/solr/WritingDistributedSearchComponents and looking though the FacetComponent, DebugComponent, StatsComponent, and HighlightComponent code. Any help figuring out why the other methods are not being called is greatly appreciated. Please ignore the println statments, they are for debug only and will be removed in the finalized, working patch.

Thanks!

I got the previous patch working. It was we close. I attached the java file and a patch for just the TermsComponent

I have cleaned up the patch, tested it against the latest version of trunk and wrote some unit tests. This patch invalidates SOLR-1156 because it also includes sorting support.

This patch includes SOLR-1156

1. Based on Matt's patch
2. Synced to trunk
3. Uses BaseDistributedTestCase

All tests pass.

I had to change TermData#frequency to an int to match the output of distributed and non-distributed cases. It is theoretically possible to have the sum of frequencies from all shards to exceed size of an int but I don't think it is practical right now. The problem is that we represent frequency as int everywhere for non-distributed responses. If we want longs in distributed search responses then we must start using longs in non-distributed responses as well to maintain compatibility.

Matt – There is an issue open for adding SolrJ support for TermsComponent - SOLR-1139. Is it possible to replace the TermsHelper and TermData classes by classes in SOLR-1139? I'd like to have the same classes parsing responses in Solrj and distributed search.

The facet component internally uses long to add up distributed facet counts, and then uses this code:

// use <int> tags for smaller facet counts (better back compatibility)
  private Number num(long val) {
   if (val < Integer.MAX_VALUE) return (int)val;
   else return val;
  }

Yes, it's not ideal to switch from <int> to <long> in a running application, but I think it's better than failing or overflowing the int.
Client code in SolrJ should be written to handle either via ((Number)x).longValue()

Thanks for the update Yonik! I will see if I can get this and SOLR-1139 using the same classes.

Here is an updated patch that includes Shalin's suggestions:

• replace TermData with TermsResponse.Term
• updates TermsHelper to use the parsing code from TermsResponse

I also changed TermsResponse.Term#frequency to a long so that we don't overflow when calculating the frequency. Then to keep back-compatbility with existing code I do the following when writing it to the NamedList:

if (tc.getFrequency() >= freqmin && tc.getFrequency() <= freqmax) {
fieldterms.add(tc.getTerm(), ((Number)tc.getFrequency()).intValue());
cnt++;
}

Is this a good approach?

This new patch includes SOLR-1139.

The latest version of this patch includes SOLR-1139 since we are reusing code from that patch.

Thanks Matt. Can you please attach the relevant portions to SOLR-1139. We can commit SOLR-1139 first and then resolve this one.

The latest SOLR-1139 patch is included inside the latest patch I attached to this ticket. Should I separate them?

The latest SOLR-1139 patch is included inside the latest patch I attached to this ticket. Should I separate them?

Yes. I'll commit SOLR-1139 first so remove those classes from the current patch.

PS: I'm sorry if I am confusing you. It is 3AM here and I'm a little confused myself

New patch that DOES NOT include the code for SOLR-1139. Make sure you have SOLR-1139 applied before using this patch.

The latest version of this patch depends on SOLR-1139 being applied.

if (tc.getFrequency() >= freqmin && tc.getFrequency() <= freqmax) {
  fieldterms.add(tc.getTerm(), ((Number)tc.getFrequency()).intValue()); cnt++; 
}

I changed freqmin and freqmax to long and used Yonik's method to write int if possible or else switch to longs in the response.

I'll commit this shortly.

Committed revision 890199.

Thanks Matt!

Correcting Fix Version based on CHANGES.txt, see this thread for more details...

http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E

Bulk close for 3.1.0 release