Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-1547

[JAVA] Fix 8x memory over-allocation in BitVector

    Details

      Description

      Typically there are 3 ways of specifying the amount of memory needed for vectors.

      CASE (1) allocateNew() – here the application doesn't really specify the size of memory or value count. Each vector type has a default value count (4096) and therefore a default size (in bytes) is used in such cases.

      For example, for a 4 byte fixed-width vector, we will allocate 32KB of memory for a call to allocateNew().

      CASE (2) setInitialCapacity(count) followed by allocateNew() - In this case also the application doesn't specify the value count or size in allocateNew(). However, the call to setInitialCapacity() dictates the amount of memory the subsequent call to allocateNew() will allocate.

      For example, we can do setInitialCapacity(1024) and the call to allocateNew() will allocate 4KB of memory for the 4 byte fixed-width vector.

      CASE (3) allocateNew(count) - The application is specific about requirements.

      For nullable vectors, the above calls also allocate the memory for validity vector.

      The problem is that Bit Vector uses a default memory size in bytes of 4096. In other words, we allocate a vector for 4096*8 value count.

      In the default case (as explained above), the vector types have a value count of 4096 so we need only 4096 bits (512 bytes) in the bit vector and not really 4096 as the size in bytes.

      This happens in CASE 1 where the application depends on the default memory allocation . In such cases, the size of buffer for bit vector is 8x than actually needed.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/arrow/pull/1109

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/arrow/pull/1109
          Hide
          wesmckinn Wes McKinney added a comment -

          Issue resolved by pull request 1109
          https://github.com/apache/arrow/pull/1109

          Show
          wesmckinn Wes McKinney added a comment - Issue resolved by pull request 1109 https://github.com/apache/arrow/pull/1109
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user wesm commented on the issue:

          https://github.com/apache/arrow/pull/1109

          +1

          Show
          githubbot ASF GitHub Bot added a comment - Github user wesm commented on the issue: https://github.com/apache/arrow/pull/1109 +1
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user siddharthteotia commented on the issue:

          https://github.com/apache/arrow/pull/1109

          cc @jacques-n , @StevenMPhillips

          Show
          githubbot ASF GitHub Bot added a comment - Github user siddharthteotia commented on the issue: https://github.com/apache/arrow/pull/1109 cc @jacques-n , @StevenMPhillips
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user siddharthteotia opened a pull request:

          https://github.com/apache/arrow/pull/1109

          ARROW-1547: [JAVA] Fix 8x memory over-allocation in BitVector

          Problem:

          Typically there are 3 ways of specifying the amount of memory needed for vectors.
          CASE (1) allocateNew() – here the application doesn't really specify the size of memory or value count. Each vector type has a default value count (4096) and therefore a default size (in bytes) is used in such cases.

          For example, for a 4 byte fixed-width vector, we will allocate 32KB of memory for a call to allocateNew().

          CASE (2) setInitialCapacity(count) followed by allocateNew() - In this case also the application doesn't specify the value count or size in allocateNew(). However, the call to setInitialCapacity() dictates the amount of memory the subsequent call to allocateNew() will allocate.

          For example, we can do setInitialCapacity(1024) and the call to allocateNew() will allocate 4KB of memory for the 4 byte fixed-width vector.

          CASE (3) allocateNew(count) - The application is specific about requirements.
          For nullable vectors, the above calls also allocate the memory for validity vector.

          The problem is that Bit Vector uses a default memory size in bytes of 4096. In other words, we allocate a vector for 4096*8 value count.

          In the default case (as explained above), the vector types have a value count of 4096 so we need only 4096 bits (512 bytes) in the bit vector and not really 4096 as the size in bytes.

          This happens in CASE 1 where the application depends on the default memory allocation . In such cases, the size of buffer for bit vector is 8x than actually needed

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/siddharthteotia/arrow ARROW-1547

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/arrow/pull/1109.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #1109


          commit f3d123484e95dfec4d5d83a57de85696aac2cd46
          Author: siddharth <siddharth@dremio.com>
          Date: 2017-09-17T23:16:39Z

          ARROW-1547: Fix 8x memory over-allocation in BitVector


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user siddharthteotia opened a pull request: https://github.com/apache/arrow/pull/1109 ARROW-1547 : [JAVA] Fix 8x memory over-allocation in BitVector Problem: Typically there are 3 ways of specifying the amount of memory needed for vectors. CASE (1) allocateNew() – here the application doesn't really specify the size of memory or value count. Each vector type has a default value count (4096) and therefore a default size (in bytes) is used in such cases. For example, for a 4 byte fixed-width vector, we will allocate 32KB of memory for a call to allocateNew(). CASE (2) setInitialCapacity(count) followed by allocateNew() - In this case also the application doesn't specify the value count or size in allocateNew(). However, the call to setInitialCapacity() dictates the amount of memory the subsequent call to allocateNew() will allocate. For example, we can do setInitialCapacity(1024) and the call to allocateNew() will allocate 4KB of memory for the 4 byte fixed-width vector. CASE (3) allocateNew(count) - The application is specific about requirements. For nullable vectors, the above calls also allocate the memory for validity vector. The problem is that Bit Vector uses a default memory size in bytes of 4096. In other words, we allocate a vector for 4096*8 value count. In the default case (as explained above), the vector types have a value count of 4096 so we need only 4096 bits (512 bytes) in the bit vector and not really 4096 as the size in bytes. This happens in CASE 1 where the application depends on the default memory allocation . In such cases, the size of buffer for bit vector is 8x than actually needed You can merge this pull request into a Git repository by running: $ git pull https://github.com/siddharthteotia/arrow ARROW-1547 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/arrow/pull/1109.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1109 commit f3d123484e95dfec4d5d83a57de85696aac2cd46 Author: siddharth <siddharth@dremio.com> Date: 2017-09-17T23:16:39Z ARROW-1547 : Fix 8x memory over-allocation in BitVector

            People

            • Assignee:
              siddteotia Siddharth Teotia
              Reporter:
              siddteotia Siddharth Teotia
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development