[TIKA-493] Support for macro languages

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.7
• Component/s: languageidentifier
• Assignee: Kenneth William Krugler
• Reporter: Jan Høydahl
• Created: 21/Aug/10 18:11
• Updated: 04/Feb/16 20:30

Description

Some languages have variants, and there are ISO codes to identify both the variants as well as a code to identify the macro-language. There should be a way to tell whether the identified language is part of a "macro language" and to return the macro language. This is because different applications require different codes. E.g. for search it makes sense to tag the document with both the unique code and the macro code.

Example:
Norwegian: no
Norwegian bokmål: nb
Norwegian nynorsk: nn

The getLanguage() call should continue to return the most correct and specific ISO code (according to which language profile matched).

In addition, it should be possible to get the macro language.

Proposed implementation:
Add some new methods:

public boolean hasMacroLanguage() // true | false
public String getMacroLanguage() // In case of "nn" or "nb", result would be "no"

The definition of macro languages can be added in the property file introduced in TIKA-490.

Issue Links

• relates to: TIKA-1723 Integrate language-detector into Tika