[TIKA-4278] TextAndCSVParser doesn't detect semicolon separated file - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 2.9.2
Fix Version/s: 3.0.0, 2.9.3
Component/s: parser
Labels:
- csv
- csvparser

External issue URL:
https://stackoverflow.com/questions/78672800/tika-2-x-unable-to-detect-csv-correctly-for-excel-output-format-semicolon-sep

Description

I ran the code from the attached SO issue and yes it doesn't detect semicolon separated files. The reason is this line in TextAndCSVParser.java:

private static final char[] DEFAULT_DELIMITERS = new char[]\{',', '\t'};

This is later used by CSVSniffer. For some reason the other delimiters (pipe, colon and semicolon) aren't in that array, although they are in CHAR_TO_STRING_DELIMITER_MAP. I modified DEFAULT_DELIMITERS and now it works for semicolon.

Can I change this by adding the missing delimiters or was there a reason that I missed? Proposed change would change CSVSniffer so that delimiters is a set and then pass CHAR_TO_STRING_DELIMITER_MAP.keySet().