[SOLR-13356] Language detection per value

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: contrib - LangId
• Labels: UpdateProcessor detection language
• Assignee: Unassigned
• Reporter: Marco Remy
• Created: 29/Mar/19 14:42
• Updated: 08/Jun/19 14:58

Description

Hello,

We are using the LangDetect language detection processor with individual field mapping.

<processor class="solr.LangDetectLanguageIdentifierUpdateProcessorFactory">
  ...
  <bool name="langid.map">true</bool>
  <bool name="langid.map.individual">true</bool>
</processor>

If a (simple structured) document is indexed containing different languages in a multivalued field, only one language will be predicted.

eg:

<doc>
  <field>This is any text</field>
  <field>Das ist irgendein Text</field>
</doc>

The result will be either field_en or field_de and both values are mapped into that localized field. In effect some values won't be analyzed properly according to their actual language.

As enhancement, the detection should be available per value on multivalued fields. So their values can be mapped individually.