Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-3519

A new column page MemoryBlock is allocated at each row addition to table page if having string column with local dictionary enabled.

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None

      Description

      Context:

      For a string column with local dictionary enabled, a column page of

      `UnsafeFixLengthColumnPage` with datatype `DataTypes.BYTE_ARRAY` is created for `encodedPage` along with regular `actualPage` of `UnsafeVarLengthColumnPage`. 

      We have `capacity` field in 

       the `UnsafeFixLengthColumnPage`. And this field indicates the capacity of  allocated

      `memoryBlock` for the page. `ensureMemory()` method is being called while adding rows to check if 

      `totalLength + requestSize > capacity` to allocate a new memoryBlock if there is no room to add the next row, copy the old context(prev rows) and free the old memoryBlock.

       

      Issue:

      While, `UnsafeFixLengthColumnPage` with with datatype `DataTypes.BYTE_ARRAY` is created for `encodedPage`, we have not assigned the `capacity` field with allocated memory block size. Hence, when we add a row to tablePage, ensureMemory() check always fails, allocates a new column page memoryBlock, copy the old context(prev rows) and free the old memoryBlock. This allocation of new memoryBlock and free of old memoryBlock happens at row addition for the string columns with local dictionary enabled.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              VenuReddy Venugopal Reddy K
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: