[TIKA-4278] TextAndCSVParser doesn't detect semicolon separated file

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.2
• Fix Version/s: 3.0.0, 2.9.3
• Component/s: parser
• Labels: csv csvparser
• External issue URL: https://stackoverflow.com/questions/78672800/tika-2-x-unable-to-detect-csv-correctly-for-excel-output-format-semicolon-sep
• Assignee: Unassigned
• Reporter: Tilman Hausherr
• Created: 13/Jul/24 10:57
• Updated: 17/Oct/24 03:47
• Resolved: 17/Oct/24 03:47

Description

I ran the code from the attached SO issue and yes it doesn't detect semicolon separated files. The reason is this line in TextAndCSVParser.java:

private static final char[] DEFAULT_DELIMITERS = new char[]\{',', '\t'};

This is later used by CSVSniffer. For some reason the other delimiters (pipe, colon and semicolon) aren't in that array, although they are in CHAR_TO_STRING_DELIMITER_MAP. I modified DEFAULT_DELIMITERS and now it works for semicolon.

Can I change this by adding the missing delimiters or was there a reason that I missed? Proposed change would change CSVSniffer so that delimiters is a set and then pass CHAR_TO_STRING_DELIMITER_MAP.keySet().

Issue Links

• is duplicated by: TIKA-3155 Parse Error while extracting CSV files
• links to: GitHub Pull Request #1976