Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-198

[Java] OutOfMemoryError for vector test case

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Resolved
    • Affects Version/s: None
    • Fix Version/s: 0.10.0
    • Component/s: Java
    • Labels:
      None

      Description

      When executing vector component TestValueVector test cases(testVariableVectorReallocation,testFixedVectorReallocation),exception can be seen.

      Exception:
      java.lang.Exception: Unexpected exception, expected<org.apache.arrow.vector.util.OversizedAllocationException> but was<org.apache.arrow.memory.OutOfMemoryException>

      where the issue comes from:
      public void testFixedVectorReallocation() {
      ....
      try {
      vector.allocateNew(expectedValueCapacity); <-- where the exception comes from
      ....

      analysis:

      allocateNew will invoke classes from io.netty.buffer from memory component and finally invoke netty third party package.I assume there is no issue when invoking memory part.Then I trace that the issue is caused by oversizing value of initialCapacity setting when invoking AbstractByteBufAllocator class in third party netty-buffer-4.0.27.final.jar. I create a test case which can reproduces the issue. value of parameter initialCapacity and maxCapacity are obtained from debugging procedure

      public void testAllocator(){
      int initialCapacity=2147483644;
      int maxCapacity=2147483647;
      ByteBuf buf = UnpooledByteBufAllocator.DEFAULT.directBuffer(initialCapacity, maxCapacity);
      }

      And I checked the rooted affected class java.nio.Bits:

      static void reserveMemory(long size, int cap) {
      .....
      if (totalCapacity + cap > maxMemory)
      throw new OutOfMemoryError("Direct buffer memory");
      .....

      maxMemory can only be set to 1836056576 at most,but cap(initialCapacity) was 2147483644 at this moment.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                lfz_carlos Fangzhou
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: