[CARBONDATA-3761] Remove redundant conversion for complex type insert

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Assignee: Ajantha Bhat
• Reporter: Ajantha Bhat
• Created: 02/Apr/20 04:56
• Updated: 10/Apr/20 13:44
• Resolved: 10/Apr/20 13:42

Description

Problem

1. In PrimitiveDataType#writeByteArray
   DataTypeUtil.parseValue(*input.toString()*, carbonDimension)
   Here we convert every complex child element to string and then parse as an object to handle bad records. Which leads to heavy GC

1. DatatypeUtil#getBytesDataDataTypeForNoDictionaryColumn -> double,float, byte, decimal case is missing. so we convert them to string and then convert to bytes. which create more redundant objects

Solution

1. For new Insert into flow, no need to handle bad records for complex types as it is already validated in source table. So, use object directly. This can decrease the memory foot print for complex type insert
2. Add a case for double,float, byte, decimal data type.

Issue Links

• links to: GitHub Pull Request #3687