[TEZ-3257] Flaky TestUnorderedPartitionedKVWriter

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.2, 0.9.0, 0.8.4
• Assignee: Ming Ma
• Reporter: Ming Ma
• Created: 12/May/16 17:57
• Updated: 09/Jul/16 19:15
• Resolved: 26/May/16 18:04

Description

For example,

testSkippedPartitions[1](org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter)  Time elapsed: 0.346 sec  <<< FAILURE!
java.lang.AssertionError: null
	at org.junit.Assert.fail(Assert.java:86)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.junit.Assert.assertTrue(Assert.java:52)
	at org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTest(TestUnorderedPartitionedKVWriter.java:661)
	at org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testSkippedPartitions(TestUnorderedPartitionedKVWriter.java:196)

The reason is the spill onSuccess callback can happen between the following two lines, which means the old buffer will be added to availableBuffers in time and thus there is no need to initialize another buffer.

      ListenableFuture<SpillResult> future = spillExecutor.submit(
          new SpillCallable(currentBuffer, codec, spilledRecordsCounter, spillPathDetails));
      Futures.addCallback(future, new SpillCallback(spillPathDetails.spillIndex));

      WrappedBuffer wb = getNextAvailableBuffer();

Can we just remove the following check in test code?

    // Verify the status of the buffers
    if (numExpectedSpills == 0) {
      assertEquals(1, kvWriter.numInitializedBuffers);
    } else {
      assertTrue(kvWriter.numInitializedBuffers > 1);
    }