patch, needs the language detection jar and its deps from revision 111 of language-detection (in the lib folder), and the profiles files (into the resources folder)

this is just for reviewing, there are a lot of svn moves etc (so i doubt you can easily apply it)

based on http://code.google.com/p/language-detection (apache license), supports 53 languages.

I've seen that, too. +1.

ok, i'd like to add this basic implementation first.

later, we should add support for some advanced parameters and refactoring:

• whitelisting should not happen in the base class as a post-filter (though this is fine as a default implementation), but subclasses should override i think. For this detector, it could improve performance.
• for this detector whitelist should support priors too (e.g. en=0.5, fr=0.1).
• we should add support for configuring smoothing parameter and maxTextLength (and, the base class's concat should respect that too).
• both this implementation and the tika implementation are copying objects across lists of language information, i think this is not very efficient to do per-document. So I think we should change the API from List<DetectedLanguage> detectLanguage() to Iterable<DetectedLanguage> detectLanguage. It seems in general it just wants the first one anyway.

Cool. The reasoning behind a list of detected languages was that a more advanced detector could go sentence by sentence and tag multi lingual documents correctly. FAST had that capability.

How does this impl compare with the Tika one for short texts? And wouldn't it make more sense to add this on the Tika level letting the detection method be configurable? Then all Tika users would benefit from it.

How does this impl compare with the Tika one for short texts? And wouldn't it make more sense to add this on the Tika level letting the detection method be configurable? Then all Tika users would benefit from it.

I have no idea, probably not that great? But i didnt compare to tika.
regarding short texts: http://shuyo.wordpress.com/2011/09/29/langdetect-is-updatedadded-profiles-of-estonian-lithuanian-latvian-slovene-and-so-on/

And wouldn't it make more sense to add this on the Tika level letting the detection method be configurable? Then all Tika users would benefit from it.

If someone wants to do this, then we can remove this implementation at that time. But for lucene/solr, I am able to commit to this project, and I think that its important for langid to be pluggable to different implementations.

For example, maybe someone ports google's detector (http://src.chromium.org/viewvc/chrome/trunk/src/third_party/cld/) to java and we expose that too, which might be interesting for short texts.

Sure, it's way better to get stuff done than debate on details Great work. Stuff can "bubble down" to Tika later just has stuff has bubbled down from Solr to Lucene..

Its not really the same in my opinion. Anyone can commit to both lucene and solr so we can always put things in the correct place.

I meant to compare with the situation before the Solr+Lucene merge. It takes a whole lot longer time to wait for a dependency to get released before you can consume it, so then it's ok to add it higher up as a first step.

Bulk close after 3.5 is released