Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-3761

Remove redundant conversion for complex type insert

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: None
    • Labels:
      None

      Description

      Problem

      1. In PrimitiveDataType#writeByteArray
        DataTypeUtil.parseValue(*input.toString()*, carbonDimension)
        Here we convert every complex child element to string and then parse as an object to handle bad records. Which leads to heavy GC
      1. DatatypeUtil#getBytesDataDataTypeForNoDictionaryColumn -> double,float, byte, decimal case is missing. so we convert them to string and then convert to bytes. which create more redundant objects

      Solution

      1. For new Insert into flow, no need to handle bad records for complex types as it is already validated in source table. So, use object directly. This can decrease the memory foot print for complex type insert
      2. Add a case for double,float, byte, decimal data type.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Ajantha_Bhat Ajantha Bhat
                Reporter:
                Ajantha_Bhat Ajantha Bhat
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h
                  3h