[SOLR-11774] langid.map.individual won't work with langid.map.keepOrig

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 5.0
• Fix Version/s: 8.0
• Component/s: contrib - LangId
• Labels: backward-incompatible
• Assignee: Jan Høydahl
• Reporter: Marco Remy
• Created: 18/Dec/17 17:27
• Updated: 02/Oct/19 17:30
• Resolved: 03/Jan/19 17:31

Description

Tried to get language detection to work.

Setting:

<processor class="org.apache.solr.update.processor.LangDetectLanguageIdentifierUpdateProcessorFactory">
      <str name="langid.fl">title,author</str>
      <str name="langid.langsField">detected_languages</str>
      <str name="langid.whitelist">de,en</str>
      <str name="langid.fallback">txt</str>
      <bool name="langid.map">true</bool>
      <bool name="langid.map.individual">true</bool>
      <bool name="langid.map.keepOrig">true</bool>
    </processor>

Main purpose

• Map fields individually
• Keep the original field

But the fields won't be mapped individually. They are mapped to a single detected language. After some hours of investigation i finally found the reason: The option langid.map.keepOrig breaks the individual mapping function. Only if it is disabled the fields will be mapped as expected.

• Regards

Issue Links

• is caused by: SOLR-3881 frequent OOM in LanguageIdentifierUpdateProcessor
• links to: GitHub Pull Request #336
• links to: GitHub Pull Request #336