[TEZ-3664] Flaky tests due to writing to /tmp directory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0, 0.9.3
• Assignee: Jonathan Turner Eagles
• Reporter: Yesha Vora
• Created: 21/Mar/17 00:27
• Updated: 24/Feb/20 22:27
• Resolved: 24/Feb/20 22:27

Description

testSingleSpill_WithPipelinedShuffle[0] unit test fails intermittently

Stacktrace

java.lang.AssertionError: null
	at org.junit.Assert.fail(Assert.java:86)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.junit.Assert.assertTrue(Assert.java:52)
	at org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.baseTestWithPipelinedTransfer(TestUnorderedPartitionedKVWriter.java:503)
	at org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter.testSingleSpill_WithPipelinedShuffle(TestUnorderedPartitionedKVWriter.java:414)
Standard Output

2017-03-20 19:50:52,538 INFO  [main] writers.TestUnorderedPartitionedKVWriter (TestUnorderedPartitionedKVWriter.java:setup(124)) - Setup. Using test dir: /tmp/TestUnorderedPartitionedKVWriter
2017-03-20 19:50:53,065 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-03-20 19:50:53,856 INFO  [Thread-5] counters.Limits (Limits.java:ensureInitialized(60)) - Counter limits initialized with parameters:  GROUP_NAME_MAX=256, MAX_GROUPS=500, COUNTER_NAME_MAX=64, MAX_COUNTERS=1200
2017-03-20 19:50:53,861 INFO  [Thread-5] writers.BaseUnorderedPartitionedKVWriter (BaseUnorderedPartitionedKVWriter.java:<init>(148)) - Instantiating Partitioner: [org.apache.tez.runtime.library.common.writers.TestUnorderedPartitionedKVWriter$PartitionerForTest]
2017-03-20 19:50:53,870 INFO  [Thread-5] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:<init>(221)) - destinationVertexName: numBuffers=2, sizePerBuffer=1024, skipBuffers=false, pipelinedShuffle=true, numPartitions=10
2017-03-20 19:50:53,871 INFO  [Thread-5] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:setupNextBuffer(333)) - destinationVertexName: Moving to next buffer and triggering spill
2017-03-20 19:50:53,913 INFO  [UnorderedOutSpiller {destinationVertexName}] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:callInternal(432)) - destinationVertexName: Finished spill 0
2017-03-20 19:50:53,934 INFO  [Thread-5] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) - destinationVertexName: Adding spill event for spill (final update=false), spillId=0
2017-03-20 19:50:53,938 INFO  [Thread-5] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:close(473)) - destinationVertexName: Waiting for all spills to complete : Pending : 0
2017-03-20 19:50:53,938 INFO  [Thread-5] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:close(492)) - destinationVertexName: All spills complete
2017-03-20 19:50:53,948 INFO  [Thread-5] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:callInternal(432)) - destinationVertexName: Finished spill 1
2017-03-20 19:50:53,959 INFO  [Thread-5] writers.UnorderedPartitionedKVWriter (UnorderedPartitionedKVWriter.java:sendPipelinedEventForSpill(914)) - destinationVertexName: Adding spill event for spill (final update=true), spillId=1
2017-03-20 19:50:53,972 INFO  [main] writers.TestUnorderedPartitionedKVWriter (TestUnorderedPartitionedKVWriter.java:cleanup(132)) - CleanUp