Description
For example,
testSkippedPartitions[1](org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter) Time elapsed: 0.346 sec <<< FAILURE! java.lang.AssertionError: null at org.junit.Assert.fail(Assert.java:86) at org.junit.Assert.assertTrue(Assert.java:41) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTest(TestUnorderedPartitionedKVWriter.java:661) at org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testSkippedPartitions(TestUnorderedPartitionedKVWriter.java:196)
The reason is the spill onSuccess callback can happen between the following two lines, which means the old buffer will be added to availableBuffers in time and thus there is no need to initialize another buffer.
ListenableFuture<SpillResult> future = spillExecutor.submit( new SpillCallable(currentBuffer, codec, spilledRecordsCounter, spillPathDetails)); Futures.addCallback(future, new SpillCallback(spillPathDetails.spillIndex)); WrappedBuffer wb = getNextAvailableBuffer();
Can we just remove the following check in test code?
// Verify the status of the buffers if (numExpectedSpills == 0) { assertEquals(1, kvWriter.numInitializedBuffers); } else { assertTrue(kvWriter.numInitializedBuffers > 1); }