[NIFI-4882] CSVRecordReader should utilize specified date/time/timestamp format at its convertSimpleIfPossible method

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.6.0
• Component/s: Extensions
• Assignee: Derek Straka
• Reporter: Koji Kawamura
• Created: 15/Feb/18 09:29
• Updated: 26/Mar/18 19:55
• Resolved: 22/Mar/18 01:59

Description

CSVRecordReader.convertSimpleIfPossible method is used by ValidateRecord. The method does not coerce values to the target schema field type if the raw string representation in the input CSV file is not compatible.

The type compatibility check is implemented as follows. But it does not use user specified date/time/timestamp format:

                // This will return 'false' for input '01/01/1900' when user specified custom format 'MM/dd/YYYY'
                if (DataTypeUtils.isCompatibleDataType(trimmed, dataType)) {
                    // The LAZY_DATE_FORMAT should be used to check compatibility, too.
                    return DataTypeUtils.convertType(trimmed, dataType, LAZY_DATE_FORMAT, LAZY_TIME_FORMAT, LAZY_TIMESTAMP_FORMAT, fieldName);
                } else {
                    return value;
                }

If input date strings have different format than the default format 'yyyy-MM-dd', then ValidateRecord processor can not validate input records.

JacksonCSVRecordReader has the identical methods with CSVRecordReader. Those classes should have an abstract class.

Issue Links

• contains: NIFI-4509 Validate date format in records
• relates to: NIFI-4983 JsonTreeReader and JsonPathReader should utilize specified date/time/timestamp format when used from ValidateRecord
• links to: GitHub Pull Request #2473