[TIKA-3516] Unexpected charset IBM424_rtl detected for utf_8 file by CharsetDetector

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: detector, parser
• Assignee: Tim Allison
• Reporter: Chaitra Rajappa
• Created: 06/Aug/21 17:41
• Updated: 22/May/22 13:23
• Resolved: 17/Aug/21 15:21

Description

Hi,
The CharsetDetector detects the wrong charset for a file as IBM424_rtl. 
Resulting in exception 

java.nio.charset.UnsupportedCharsetException: IBM424_rtl 17 at java.nio.charset.Charset.forName(Charset.java:531)

I see there is also an existing ticket with the same issue thats not been fixed.

https://issues.apache.org/jira/browse/TIKA-2396
Please suggest the changes to fix this. 

Versions being used:

apache-core - 1.20
apache-parsers-1.20

Thanks

Issue Links

• depends upon: TIKA-3525 Allow users to configure skipping of unsupported charsets in charset detection
• is related to: TIKA-1236 CharsetDetector returning unsupported encoding for some 7-bit Outlook/MSG files