[DRILL-4112] BaseDataValueVector.getBuffers(false) doesn't return it's inner buffer which affects transfer of ownership between allocators in ExternalSort - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Open
Priority: Major
Resolution: Unresolved
Affects Version/s: 1.2.0
Fix Version/s: None
Component/s: None
Labels:
None

Description

As part of fixing ~~DRILL-2274~~, to avoid running out of memory in sort when it spilled to disk we close the copierAllocator after we receive the last batch from upstream, this way it releases all it's allocated memory back to it's parent allocator. The assumption is that the copierAllocator doesn't own any buffers once a mergeAndSpill is done and we no longer need it if we are no longer
spilling to disk.

This causes it's accountor to complain that some buffers weren't closed (when assertions are enabled), which goes against the previous assumption.

The copierAllocator is used to allocate many batches while spilling but only one is kept in memory and the following method transfers it's ownership to the operator's allocator:

  private void takeOwnership(VectorAccessible batch) {
    for (VectorWrapper<?> w : batch) {
      DrillBuf[] bufs = w.getValueVector().getBuffers(false);
      for (DrillBuf buf : bufs) {
        if (buf.isRootBuffer()) {
          oContext.getAllocator().takeOwnership(buf);
        }
      }
    }
  }

The problem comes from how BaseDataValueVector.getBuffers(boolean clear) is implemented:

  @Override
  public DrillBuf[] getBuffers(boolean clear) {
    DrillBuf[] out;
    if (getBufferSize() == 0) {
      out = new DrillBuf[0];
    } else {
      out = new DrillBuf[]{data};
      if (clear) {
        data.readerIndex(0);
        data.retain(1);
      }
    }
    if (clear) {
      clear();
    }
    return out;
  }

If we don't write any value into the vector, and call getBuffers(false), it won't return it's allocated buffer but an empty array instead. Thus, the buffer won't be transferred to the operator's allocator.

The following unit test will exposes the problem:

  @Test
  public void testTakeOwnership() throws Exception {
    final MaterializedField field = MaterializedField.create(SchemaPath.getSimplePath(""), NullableVarCharHolder.TYPE);
    final DrillConfig drillConfig = DrillConfig.create();
    final BufferAllocator allocator = RootAllocatorFactory.newRoot(drillConfig);
    final BufferAllocator childAllocator = allocator.getChildAllocator(null, 10000, 20000, false);

    final NullableVarCharVector vector = new NullableVarCharVector(field, childAllocator);
    vector.allocateNew(1024 * 10, 1024);

    DrillBuf[] buffers = vector.getBuffers(false);
    for (final DrillBuf buffer : buffers) {
      allocator.takeOwnership(buffer);
    }

    childAllocator.close();
    allocator.close();
  }

When closing childAllocator we get the following exception (when assertions are enabled):

java.lang.IllegalStateException: Attempted to close accountor with 3 buffer(s) still allocated.


	Total 1 allocation(s) of byte size(s): 4100, at stack location:
		org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:256)
		org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:209)
		org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:191)
		org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:386)
		org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:224)
		org.apache.drill.exec.record.vector.TestValueVector.testTakeOwnership(TestValueVector.java:123)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		java.lang.reflect.Method.invoke(Method.java:606)
		org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
		org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:120)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
		mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		java.lang.reflect.Method.invoke(Method.java:606)
		mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
		mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
		mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
		org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
		org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
		org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
		org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)


	Total 1 allocation(s) of byte size(s): 1024, at stack location:
		org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:256)
		org.apache.drill.exec.vector.UInt1Vector.allocateBytes(UInt1Vector.java:209)
		org.apache.drill.exec.vector.UInt1Vector.allocateNew(UInt1Vector.java:191)
		org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:225)
		org.apache.drill.exec.record.vector.TestValueVector.testTakeOwnership(TestValueVector.java:123)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		java.lang.reflect.Method.invoke(Method.java:606)
		org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
		org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:120)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
		mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		java.lang.reflect.Method.invoke(Method.java:606)
		mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
		mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
		mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
		org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
		org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
		org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
		org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)


	Total 1 allocation(s) of byte size(s): 10240, at stack location:
		org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:256)
		org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:385)
		org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:224)
		org.apache.drill.exec.record.vector.TestValueVector.testTakeOwnership(TestValueVector.java:123)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		java.lang.reflect.Method.invoke(Method.java:606)
		org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
		org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:120)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
		mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		java.lang.reflect.Method.invoke(Method.java:606)
		mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
		mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
		mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
		org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
		org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
		org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
		org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
java.lang.IllegalStateException: Attempted to close accountor with 3 buffer(s) still allocated.


	Total 1 allocation(s) of byte size(s): 4100, at stack location:
		org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:256)
		org.apache.drill.exec.vector.UInt4Vector.allocateBytes(UInt4Vector.java:209)
		org.apache.drill.exec.vector.UInt4Vector.allocateNew(UInt4Vector.java:191)
		org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:386)
		org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:224)
		org.apache.drill.exec.record.vector.TestValueVector.testTakeOwnership(TestValueVector.java:123)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
		org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:120)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
		mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
		mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
		mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
		org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
		org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
		org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
		org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)


	Total 1 allocation(s) of byte size(s): 1024, at stack location:
		org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:256)
		org.apache.drill.exec.vector.UInt1Vector.allocateBytes(UInt1Vector.java:209)
		org.apache.drill.exec.vector.UInt1Vector.allocateNew(UInt1Vector.java:191)
		org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:225)
		org.apache.drill.exec.record.vector.TestValueVector.testTakeOwnership(TestValueVector.java:123)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
		org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:120)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
		mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
		mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
		mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
		org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
		org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
		org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
		org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)


	Total 1 allocation(s) of byte size(s): 10240, at stack location:
		org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.buffer(TopLevelAllocator.java:256)
		org.apache.drill.exec.vector.VarCharVector.allocateNew(VarCharVector.java:385)
		org.apache.drill.exec.vector.NullableVarCharVector.allocateNew(NullableVarCharVector.java:224)
		org.apache.drill.exec.record.vector.TestValueVector.testTakeOwnership(TestValueVector.java:123)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
		org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.executeTestMethod(JUnit4TestRunnerDecorator.java:120)
		mockit.integration.junit4.internal.JUnit4TestRunnerDecorator.invokeExplosively(JUnit4TestRunnerDecorator.java:65)
		mockit.integration.junit4.internal.MockFrameworkMethod.invokeExplosively(MockFrameworkMethod.java:29)
		sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
		sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		mockit.internal.util.MethodReflection.invokeWithCheckedThrows(MethodReflection.java:95)
		mockit.internal.annotations.MockMethodBridge.callMock(MockMethodBridge.java:76)
		mockit.internal.annotations.MockMethodBridge.invoke(MockMethodBridge.java:41)
		org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java)
		org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
		org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
		org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
		org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)

	at org.apache.drill.exec.memory.Accountor.close(Accountor.java:381)
	at org.apache.drill.exec.memory.TopLevelAllocator$ChildAllocator.close(TopLevelAllocator.java:327)
	at org.apache.drill.exec.record.vector.TestValueVector.testTakeOwnership(TestValueVector.java:130)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

Attachments

Activity

People

Assignee:: Unassigned

Reporter:: Abdel Hakim Deneche

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 18/Nov/15 18:51

Updated:: 18/Nov/15 19:43