[FLINK-29689] NIFI Performance issue - ConvertExcelToCSVProcessor to handle more data

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Component/s: API / DataSet
• Labels: CSV excel nifi poi
• Assignee: Unassigned
• Reporter: Vrinda Palod
• Created: 19/Oct/22 12:14
• Updated: 19/Oct/22 12:36
• Resolved: 19/Oct/22 12:36

Description

Currently , when we try to convert an excel with records more than 100,000,000 record type, it throws error and doesnt convert the file to csv format.

 

This can be fixed by adding the below line before reading the excel in the code. 

IOUtils.setByteArrayMaxOverride(2147483647);

 

The value hardcoded is the max length of int array above which we cant handle for now. 

 

Issue Links

• is a clone of: FLINK-10684 Improve the CSV reading process