[MAPREDUCE-7262] MRApp helpers block for long intervals (500ms) - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Minor
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 3.3.0, 3.1.4, 3.2.2, 2.10.1
Component/s: mr-am
Labels:
None

Description

MRApp has a set of methods used as helpers in test cases such as: waitForInternalState(TA), waitForState(TA), waitForState(Job)..etc

When the condition fails, the thread sleeps for a minimum of 500ms before rechecking the new state of the Job/TA.
Example:

  public void waitForState(Task task, TaskState finalState) throws Exception {
    int timeoutSecs = 0;
    TaskReport report = task.getReport();
    while (!finalState.equals(report.getTaskState()) &&
        timeoutSecs++ < 20) {
      System.out.println("Task State for " + task.getID() + " is : "
          + report.getTaskState() + " Waiting for state : " + finalState
          + "   progress : " + report.getProgress());
      report = task.getReport();
      Thread.sleep(500);
    }
    System.out.println("Task State is : " + report.getTaskState());
    Assert.assertEquals("Task state is not correct (timedout)", finalState,
        report.getTaskState());
  }

I suggest to reduce the interval 500 to 50, while incrementing the number of retries to 200. this will potentially make the test cases run faster. Also, the System.out calls need to be removed because they are not adding information dumping the current state on every iteration.

A tentative list of Junits affected by the change:

Method
    waitForInternalState(JobImpl, JobStateInternal)
Found usages  (12 usages found)
    org.apache.hadoop.mapreduce.v2.app  (10 usages found)
        TestJobEndNotifier  (3 usages found)
            testNotificationOnLastRetry(boolean)  (1 usage found)
                214 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
            testAbsentNotificationOnNotLastRetryUnregistrationFailure()  (1 usage found)
                256 app.waitForInternalState(job, JobStateInternal.REBOOT);
            testNotificationOnLastRetryUnregistrationFailure()  (1 usage found)
                289 app.waitForInternalState(job, JobStateInternal.REBOOT);
        TestKill  (5 usages found)
            testKillJob()  (1 usage found)
                70 app.waitForInternalState((JobImpl) job, JobStateInternal.RUNNING);
            testKillTask()  (1 usage found)
                108 app.waitForInternalState((JobImpl) job, JobStateInternal.RUNNING);
            testKillTaskWait()  (1 usage found)
                219 app.waitForInternalState((JobImpl) job, JobStateInternal.KILLED);
            testKillTaskWaitKillJobAfterTA_DONE()  (1 usage found)
                266 app.waitForInternalState((JobImpl)job, JobStateInternal.KILLED);
            testKillTaskWaitKillJobBeforeTA_DONE()  (1 usage found)
                316 app.waitForInternalState((JobImpl)job, JobStateInternal.KILLED);
        TestMRApp  (2 usages found)
            testJobSuccess()  (1 usage found)
                494 app.waitForInternalState(job, JobStateInternal.SUCCEEDED);
            testJobRebootOnLastRetryOnUnregistrationFailure()  (1 usage found)
                542 app.waitForInternalState((JobImpl) job, JobStateInternal.REBOOT);
    org.apache.hadoop.mapreduce.v2.app.rm  (2 usages found)
        TestRMContainerAllocator  (2 usages found)
            testReportedAppProgress()  (1 usage found)
                1050 mrApp.waitForInternalState((JobImpl) job, JobStateInternal.RUNNING);
            testReportedAppProgressWithOnlyMaps()  (1 usage found)
                1202 mrApp.waitForInternalState((JobImpl)job, JobStateInternal.RUNNING);


--------------------------------------------------------------------------------------

Method
    waitForState(TaskAttempt, TaskAttemptState)
Found usages  (72 usages found)
    org.apache.hadoop.mapreduce.v2  (2 usages found)
        TestSpeculativeExecutionWithMRApp  (2 usages found)
            testSpeculateSuccessfulWithoutUpdateEvents()  (1 usage found)
                212 app.waitForState(taskAttempt.getValue(), TaskAttemptState.SUCCEEDED);
            testSpeculateSuccessfulWithUpdateEvents()  (1 usage found)
                275 app.waitForState(taskAttempt.getValue(), TaskAttemptState.SUCCEEDED);
    org.apache.hadoop.mapreduce.v2.app  (67 usages found)
        TestAMInfos  (1 usage found)
            testAMInfosWithoutRecoveryEnabled()  (1 usage found)
                58 app.waitForState(taskAttempt, TaskAttemptState.RUNNING);
        TestFetchFailure  (11 usages found)
            testFetchFailure()  (3 usages found)
                77 app.waitForState(mapAttempt1, TaskAttemptState.RUNNING);
                109 app.waitForState(reduceAttempt, TaskAttemptState.RUNNING);
                130 app.waitForState(mapAttempt2, TaskAttemptState.RUNNING);
            testFetchFailureWithRecovery()  (3 usages found)
                209 app.waitForState(mapAttempt1, TaskAttemptState.RUNNING);
                230 app.waitForState(reduceAttempt, TaskAttemptState.RUNNING);
                262 app.waitForState(mapAttempt1, TaskAttemptState.RUNNING);
            testFetchFailureMultipleReduces()  (5 usages found)
                304 app.waitForState(mapAttempt1, TaskAttemptState.RUNNING);
                327 app.waitForState(reduceAttempt, TaskAttemptState.RUNNING);
                333 app.waitForState(reduceAttempt2, TaskAttemptState.RUNNING);
                338 app.waitForState(reduceAttempt3, TaskAttemptState.RUNNING);
                372 app.waitForState(mapAttempt2, TaskAttemptState.RUNNING);
        TestKill  (6 usages found)
            testKillTaskWait()  (2 usages found)
                204 app.waitForState(mapAttempt, TaskAttemptState.RUNNING);
                206 app.waitForState(reduceAttempt, TaskAttemptState.RUNNING);
            testKillTaskWaitKillJobAfterTA_DONE()  (2 usages found)
                242 app.waitForState(mapAttempt, TaskAttemptState.RUNNING);
                244 app.waitForState(reduceAttempt, TaskAttemptState.RUNNING);
            testKillTaskWaitKillJobBeforeTA_DONE()  (2 usages found)
                290 app.waitForState(mapAttempt, TaskAttemptState.RUNNING);
                292 app.waitForState(reduceAttempt, TaskAttemptState.RUNNING);
        TestMRApp  (7 usages found)
            testCommitPending()  (3 usages found)
                114 app.waitForState(attempt, TaskAttemptState.RUNNING);
                123 app.waitForState(attempt, TaskAttemptState.COMMIT_PENDING);
                132 app.waitForState(attempt, TaskAttemptState.COMMIT_PENDING);
            testCompletedMapsForReduceSlowstart()  (2 usages found)
                169 app.waitForState(task1Attempt, TaskAttemptState.RUNNING);
                170 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
            testUpdatedNodes()  (2 usages found)
                287 app.waitForState(task1Attempt, TaskAttemptState.KILLED);
                288 app.waitForState(task2Attempt, TaskAttemptState.KILLED);
        TestMRClientService  (2 usages found)
            test()  (1 usage found)
                90 app.waitForState(attempt, TaskAttemptState.RUNNING);
            testViewAclOnlyCannotModify()  (1 usage found)
                215 app.waitForState(attempt, TaskAttemptState.RUNNING);
        TestRecovery  (40 usages found)
            testCrashed()  (8 usages found)
                165 app.waitForState(task1Attempt1, TaskAttemptState.RUNNING);
                166 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
                176 app.waitForState(task1Attempt1, TaskAttemptState.FAILED);
                197 app.waitForState(task1Attempt2, TaskAttemptState.FAILED);
                210 app.waitForState(task1Attempt3, TaskAttemptState.RUNNING);
                218 app.waitForState(task1Attempt3, TaskAttemptState.KILLED);
                232 app.waitForState(task1Attempt4, TaskAttemptState.RUNNING);
                278 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
            testCrashOfMapsOnlyJob()  (4 usages found)
                382 app.waitForState(task1Attempt, TaskAttemptState.RUNNING);
                383 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
                384 app.waitForState(task3Attempt, TaskAttemptState.RUNNING);
                435 app.waitForState(task3Attempt, TaskAttemptState.RUNNING);
            testRecoverySuccessUsingCustomOutputCommitter()  (4 usages found)
                527 app.waitForState(task1Attempt, TaskAttemptState.RUNNING);
                528 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
                529 app.waitForState(task3Attempt, TaskAttemptState.RUNNING);
                581 app.waitForState(task3Attempt, TaskAttemptState.RUNNING);
            testRecoveryWithSpillEncryption()  (3 usages found)
                621 app.waitForState(mapAttempt, TaskAttemptState.RUNNING);
                647 app.waitForState(mapAttempt, TaskAttemptState.RUNNING);
                654 app.waitForState(redAttempt, TaskAttemptState.RUNNING);
            testRecoveryFailsUsingCustomOutputCommitter()  (4 usages found)
                706 app.waitForState(task1Attempt, TaskAttemptState.RUNNING);
                707 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
                708 app.waitForState(task3Attempt, TaskAttemptState.RUNNING);
                760 app.waitForState(task3Attempt, TaskAttemptState.RUNNING);
            testMultipleCrashes()  (3 usages found)
                821 app.waitForState(task1Attempt1, TaskAttemptState.RUNNING);
                822 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
                869 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
            testOutputRecovery()  (2 usages found)
                946 app.waitForState(task1Attempt1, TaskAttemptState.RUNNING);
                1013 app.waitForState(reduce2Attempt, TaskAttemptState.RUNNING);
            testOutputRecoveryMapsOnly()  (2 usages found)
                1123 app.waitForState(task1Attempt1, TaskAttemptState.RUNNING);
                1177 app.waitForState(task2Attempt1, TaskAttemptState.RUNNING);
            testRecoveryWithOldCommiter()  (2 usages found)
                1237 app.waitForState(task1Attempt1, TaskAttemptState.RUNNING);
                1304 app.waitForState(reduce2Attempt, TaskAttemptState.RUNNING);
            testSpeculative()  (5 usages found)
                1382 app.waitForState(task1Attempt1, TaskAttemptState.RUNNING);
                1383 app.waitForState(task1Attempt2, TaskAttemptState.RUNNING);
                1384 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
                1394 app.waitForState(task1Attempt1, TaskAttemptState.SUCCEEDED);
                1432 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
            testRecoveryWithoutShuffleSecret()  (3 usages found)
                1511 app.waitForState(task1Attempt, TaskAttemptState.RUNNING);
                1512 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
                1555 app.waitForState(task2Attempt, TaskAttemptState.RUNNING);
    org.apache.hadoop.mapreduce.v2.app.job.impl  (3 usages found)
        TestTaskAttempt  (3 usages found)
            verifyMillisCounters(Resource, int)  (2 usages found)
                370 app.waitForState(mta, TaskAttemptState.RUNNING);
                371 app.waitForState(rta, TaskAttemptState.RUNNING);
            testTaskAttemptAssignedKilledHistory(FailingAttemptsDuringAssignedMRApp)  (1 usage found)
                488 app.waitForState(attempt, TaskAttemptState.KILLED);

-------------------------------------------

Method
    waitForState(Task, TaskState)
Found usages  (155 usages found)
    org.apache.hadoop.mapreduce.v2  (2 usages found)
        TestSpeculativeExecutionWithMRApp  (2 usages found)
            testSpeculateSuccessfulWithoutUpdateEvents()  (1 usage found)
                179 app.waitForState(taskIter.next(), TaskState.RUNNING);
            testSpeculateSuccessfulWithUpdateEvents()  (1 usage found)
                241 app.waitForState(taskIter.next(), TaskState.RUNNING);
    org.apache.hadoop.mapreduce.v2.app  (145 usages found)
        TestAMInfos  (1 usage found)
            testAMInfosWithoutRecoveryEnabled()  (1 usage found)
                56 app.waitForState(mapTask, TaskState.RUNNING);
        TestFail  (1 usage found)
            testTaskFailWithUnusedContainer()  (1 usage found)
                192 app.waitForState(task, TaskState.SCHEDULED);
        TestFetchFailure  (18 usages found)
            testFetchFailure()  (5 usages found)
                75 app.waitForState(mapTask, TaskState.RUNNING);
                85 app.waitForState(mapTask, TaskState.SUCCEEDED);
                106 app.waitForState(reduceTask, TaskState.RUNNING);
                117 app.waitForState(mapTask, TaskState.RUNNING);
                137 app.waitForState(mapTask, TaskState.SUCCEEDED);
            testFetchFailureWithRecovery()  (6 usages found)
                207 app.waitForState(mapTask, TaskState.RUNNING);
                217 app.waitForState(mapTask, TaskState.SUCCEEDED);
                227 app.waitForState(reduceTask, TaskState.RUNNING);
                238 app.waitForState(mapTask, TaskState.RUNNING);
                260 app.waitForState(mapTask, TaskState.RUNNING);
                270 app.waitForState(mapTask, TaskState.SUCCEEDED);
            testFetchFailureMultipleReduces()  (7 usages found)
                302 app.waitForState(mapTask, TaskState.RUNNING);
                312 app.waitForState(mapTask, TaskState.SUCCEEDED);
                322 app.waitForState(reduceTask, TaskState.RUNNING);
                323 app.waitForState(reduceTask2, TaskState.RUNNING);
                324 app.waitForState(reduceTask3, TaskState.RUNNING);
                354 app.waitForState(mapTask, TaskState.RUNNING);
                379 app.waitForState(mapTask, TaskState.SUCCEEDED);
        TestKill  (9 usages found)
            testKillTaskWait()  (3 usages found)
                201 app.waitForState(mapTask, TaskState.RUNNING);
                202 app.waitForState(reduceTask, TaskState.RUNNING);
                213 app.waitForState(mapTask, TaskState.SUCCEEDED);
            testKillTaskWaitKillJobAfterTA_DONE()  (2 usages found)
                239 app.waitForState(mapTask, TaskState.RUNNING);
                240 app.waitForState(reduceTask, TaskState.RUNNING);
            testKillTaskWaitKillJobBeforeTA_DONE()  (2 usages found)
                287 app.waitForState(mapTask, TaskState.RUNNING);
                288 app.waitForState(reduceTask, TaskState.RUNNING);
            testKillTaskAttempt()  (2 usages found)
                380 app.waitForState(task1, TaskState.SCHEDULED);
                381 app.waitForState(task2, TaskState.SCHEDULED);
        TestMRApp  (24 usages found)
            testCommitPending()  (1 usage found)
                112 app.waitForState(task, TaskState.RUNNING);
            testCompletedMapsForReduceSlowstart()  (4 usages found)
                161 app.waitForState(mapTask1, TaskState.RUNNING);
                162 app.waitForState(mapTask2, TaskState.RUNNING);
                183 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                187 app.waitForState(reduceTask, TaskState.RUNNING);
            testUpdatedNodes()  (16 usages found)
                233 app.waitForState(mapTask1, TaskState.RUNNING);
                234 app.waitForState(mapTask2, TaskState.RUNNING);
                257 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                258 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                310 app.waitForState(mapTask1, TaskState.RUNNING);
                311 app.waitForState(mapTask2, TaskState.RUNNING);
                325 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                326 app.waitForState(mapTask2, TaskState.RUNNING);
                362 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                363 app.waitForState(mapTask2, TaskState.RUNNING);
                385 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                400 app.waitForState(reduceTask1, TaskState.RUNNING);
                401 app.waitForState(reduceTask2, TaskState.RUNNING);
                410 app.waitForState(reduceTask1, TaskState.SUCCEEDED);
                416 app.waitForState(reduceTask1, TaskState.SUCCEEDED);
                425 app.waitForState(reduceTask2, TaskState.SUCCEEDED);
            testJobError()  (1 usage found)
                478 app.waitForState(task, TaskState.RUNNING);
            testJobRebootNotLastRetryOnUnregistrationFailure()  (1 usage found)
                511 app.waitForState(task, TaskState.RUNNING);
            testJobRebootOnLastRetryOnUnregistrationFailure()  (1 usage found)
                536 app.waitForState(task, TaskState.RUNNING);
        TestMRClientService  (2 usages found)
            test()  (1 usage found)
                88 app.waitForState(task, TaskState.RUNNING);
            testViewAclOnlyCannotModify()  (1 usage found)
                213 app.waitForState(task, TaskState.RUNNING);
        TestRecovery  (90 usages found)
            testCrashed()  (8 usages found)
                157 app.waitForState(mapTask1, TaskState.RUNNING);
                158 app.waitForState(mapTask2, TaskState.RUNNING);
                168 app.waitForState(reduceTask, TaskState.RUNNING);
                243 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                271 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                273 app.waitForState(mapTask2, TaskState.RUNNING);
                287 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                290 app.waitForState(reduceTask, TaskState.RUNNING);
            testCrashOfMapsOnlyJob()  (9 usages found)
                369 app.waitForState(mapTask1, TaskState.RUNNING);
                370 app.waitForState(mapTask2, TaskState.RUNNING);
                371 app.waitForState(mapTask3, TaskState.RUNNING);
                399 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                400 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                427 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                428 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                430 app.waitForState(mapTask3, TaskState.RUNNING);
                446 app.waitForState(mapTask3, TaskState.SUCCEEDED);
            testRecoverySuccessUsingCustomOutputCommitter()  (9 usages found)
                514 app.waitForState(mapTask1, TaskState.RUNNING);
                515 app.waitForState(mapTask2, TaskState.RUNNING);
                516 app.waitForState(mapTask3, TaskState.RUNNING);
                544 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                545 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                573 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                574 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                576 app.waitForState(mapTask3, TaskState.RUNNING);
                591 app.waitForState(mapTask3, TaskState.SUCCEEDED);
            testRecoveryWithSpillEncryption()  (5 usages found)
                619 app.waitForState(mapper, TaskState.RUNNING);
                624 app.waitForState(mapper, TaskState.SUCCEEDED);
                645 app.waitForState(mapper, TaskState.RUNNING);
                650 app.waitForState(mapper, TaskState.SUCCEEDED);
                657 app.waitForState(reducer, TaskState.SUCCEEDED);
            testRecoveryFailsUsingCustomOutputCommitter()  (9 usages found)
                693 app.waitForState(mapTask1, TaskState.RUNNING);
                694 app.waitForState(mapTask2, TaskState.RUNNING);
                695 app.waitForState(mapTask3, TaskState.RUNNING);
                723 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                724 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                752 app.waitForState(mapTask1, TaskState.RUNNING);
                753 app.waitForState(mapTask2, TaskState.RUNNING);
                755 app.waitForState(mapTask3, TaskState.RUNNING);
                784 app.waitForState(mapTask3, TaskState.SUCCEEDED);
            testMultipleCrashes()  (9 usages found)
                813 app.waitForState(mapTask1, TaskState.RUNNING);
                814 app.waitForState(mapTask2, TaskState.RUNNING);
                835 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                862 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                864 app.waitForState(mapTask2, TaskState.RUNNING);
                878 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                905 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                906 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                909 app.waitForState(reduceTask, TaskState.RUNNING);
            testOutputRecovery()  (8 usages found)
                939 app.waitForState(mapTask1, TaskState.RUNNING);
                955 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                960 app.waitForState(reduceTask1, TaskState.RUNNING);
                973 app.waitForState(reduceTask1, TaskState.SUCCEEDED);
                998 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                1005 app.waitForState(reduceTask1, TaskState.SUCCEEDED); 
                1007 app.waitForState(reduceTask2, TaskState.RUNNING);
                1022 app.waitForState(reduceTask2, TaskState.SUCCEEDED);
            testOutputRecoveryMapsOnly()  (7 usages found)
                1116 app.waitForState(mapTask1, TaskState.RUNNING);
                1136 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                1164 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                1170 app.waitForState(mapTask2, TaskState.RUNNING);
                1186 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                1191 app.waitForState(reduceTask1, TaskState.RUNNING);
                1204 app.waitForState(reduceTask1, TaskState.SUCCEEDED);
            testRecoveryWithOldCommiter()  (8 usages found)
                1230 app.waitForState(mapTask1, TaskState.RUNNING);
                1246 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                1251 app.waitForState(reduceTask1, TaskState.RUNNING);
                1264 app.waitForState(reduceTask1, TaskState.SUCCEEDED);
                1289 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                1296 app.waitForState(reduceTask1, TaskState.SUCCEEDED); 
                1298 app.waitForState(reduceTask2, TaskState.RUNNING);
                1313 app.waitForState(reduceTask2, TaskState.SUCCEEDED);
            testSpeculative()  (8 usages found)
                1352 app.waitForState(mapTask1, TaskState.RUNNING);
                1353 app.waitForState(mapTask2, TaskState.RUNNING);
                1386 app.waitForState(reduceTask, TaskState.RUNNING);
                1397 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                1425 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                1427 app.waitForState(mapTask2, TaskState.RUNNING);
                1441 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                1444 app.waitForState(reduceTask, TaskState.RUNNING);
            testRecoveryWithoutShuffleSecret()  (10 usages found)
                1503 app.waitForState(mapTask1, TaskState.RUNNING);
                1504 app.waitForState(mapTask2, TaskState.RUNNING);
                1514 app.waitForState(reduceTask, TaskState.RUNNING);
                1523 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                1549 app.waitForState(mapTask1, TaskState.RUNNING);
                1550 app.waitForState(mapTask2, TaskState.RUNNING);
                1564 app.waitForState(mapTask2, TaskState.SUCCEEDED);
                1567 app.waitForState(mapTask1, TaskState.RUNNING);
                1576 app.waitForState(mapTask1, TaskState.SUCCEEDED);
                1579 app.waitForState(reduceTask, TaskState.RUNNING);
    org.apache.hadoop.mapreduce.v2.app.job.impl  (3 usages found)
        TestTaskAttempt  (3 usages found)
            verifyMillisCounters(Resource, int)  (2 usages found)
                361 app.waitForState(mTask, TaskState.RUNNING);
                363 app.waitForState(rTask, TaskState.RUNNING);
            testTaskAttemptAssignedKilledHistory(FailingAttemptsDuringAssignedMRApp)  (1 usage found)
                485 app.waitForState(task, TaskState.SCHEDULED);
    org.apache.hadoop.mapreduce.v2.app.launcher  (1 usage found)
        TestContainerLauncher  (1 usage found)
            testSlowNM()  (1 usage found)
                296 app.waitForState(task, TaskState.SCHEDULED);
    org.apache.hadoop.mapreduce.v2.app.rm  (4 usages found)
        TestRMContainerAllocator  (4 usages found)
            testReportedAppProgress()  (2 usages found)
                1072 mrApp.waitForState(t, TaskState.RUNNING);
                1109 mrApp.waitForState(t, TaskState.RUNNING);
            finishTask(DrainDispatcher, MockNM, MRApp, Task)  (1 usage found)
                1152 mrApp.waitForState(task, TaskState.SUCCEEDED);
            testReportedAppProgressWithOnlyMaps()  (1 usage found)
                1221 mrApp.waitForState(t, TaskState.RUNNING);

--------------------------------------------------------------------------------------


Method
    waitForState(Job, JobState)
Found usages  (121 usages found)
    org.apache.hadoop.mapreduce.v2  (2 usages found)
        TestSpeculativeExecutionWithMRApp  (2 usages found)
            testSpeculateSuccessfulWithoutUpdateEvents()  (1 usage found)
                172 app.waitForState(job, JobState.RUNNING);
            testSpeculateSuccessfulWithUpdateEvents()  (1 usage found)
                234 app.waitForState(job, JobState.RUNNING);
    org.apache.hadoop.mapreduce.v2.app  (89 usages found)
        TestAMInfos  (2 usages found)
            testAMInfosWithoutRecoveryEnabled()  (2 usages found)
                49 app.waitForState(job, JobState.RUNNING);
                73 app.waitForState(job, JobState.RUNNING);
        TestFail  (8 usages found)
            testFailTask()  (1 usage found)
                71 app.waitForState(job, JobState.SUCCEEDED);
            testMapFailureMaxPercent()  (2 usages found)
                101 app.waitForState(job, JobState.FAILED);
                115 app.waitForState(job, JobState.SUCCEEDED);
            testReduceFailureMaxPercent()  (2 usages found)
                132 app.waitForState(job, JobState.FAILED);
                148 app.waitForState(job, JobState.SUCCEEDED);
            testTimedOutTask()  (1 usage found)
                162 app.waitForState(job, JobState.FAILED);
            testTaskFailWithUnusedContainer()  (2 usages found)
                188 app.waitForState(job, JobState.RUNNING);
                203 app.waitForState(job, JobState.FAILED);
        TestFetchFailure  (7 usages found)
            testFetchFailure()  (2 usages found)
                66 app.waitForState(job, JobState.RUNNING);
                144 app.waitForState(job, JobState.SUCCEEDED);
            testFetchFailureWithRecovery()  (3 usages found)
                198 app.waitForState(job, JobState.RUNNING);
                251 app.waitForState(job, JobState.RUNNING);
                278 app.waitForState(job, JobState.SUCCEEDED);
            testFetchFailureMultipleReduces()  (2 usages found)
                291 app.waitForState(job, JobState.RUNNING);
                396 app.waitForState(job, JobState.SUCCEEDED);
        TestJobEndNotifier  (3 usages found)
            testAbsentNotificationOnNotLastRetryUnregistrationFailure()  (2 usages found)
                253 app.waitForState(job, JobState.RUNNING);
                261 app.waitForState(job, JobState.RUNNING);
            testNotificationOnLastRetryUnregistrationFailure()  (1 usage found)
                286 app.waitForState(job, JobState.RUNNING);
        TestKill  (7 usages found)
            testKillJob()  (1 usage found)
                80 app.waitForState(job, JobState.KILLED);
            testKillTask()  (1 usage found)
                124 app.waitForState(job, JobState.SUCCEEDED);
            testKillTaskWait()  (1 usage found)
                196 app.waitForState(job, JobState.RUNNING);
            testKillTaskWaitKillJobAfterTA_DONE()  (1 usage found)
                234 app.waitForState(job, JobState.RUNNING);
            testKillTaskWaitKillJobBeforeTA_DONE()  (1 usage found)
                282 app.waitForState(job, JobState.RUNNING);
            testKillTaskAttempt()  (2 usages found)
                371 app.waitForState(job, JobState.RUNNING);
                393 app.waitForState(job, JobState.SUCCEEDED);
        TestMRApp  (19 usages found)
            testMapReduce()  (1 usage found)
                84 app.waitForState(job, JobState.SUCCEEDED);
            testZeroMaps()  (1 usage found)
                93 app.waitForState(job, JobState.SUCCEEDED);
            testZeroMapReduces()  (1 usage found)
                101 app.waitForState(job, JobState.SUCCEEDED);
            testCommitPending()  (2 usages found)
                108 app.waitForState(job, JobState.RUNNING);
                140 app.waitForState(job, JobState.SUCCEEDED);
            testCompletedMapsForReduceSlowstart()  (2 usages found)
                152 app.waitForState(job, JobState.RUNNING);
                199 app.waitForState(job, JobState.SUCCEEDED);
            testUpdatedNodes()  (3 usages found)
                226 app.waitForState(job1, JobState.RUNNING);
                353 app.waitForState(job2, JobState.RUNNING);
                440 app.waitForState(job2, JobState.SUCCEEDED);
            testJobError()  (2 usages found)
                474 app.waitForState(job, JobState.RUNNING);
                486 app.waitForState(job, JobState.ERROR);
            testJobSuccess()  (1 usage found)
                499 app.waitForState(job, JobState.SUCCEEDED);
            testJobRebootNotLastRetryOnUnregistrationFailure()  (2 usages found)
                507 app.waitForState(job, JobState.RUNNING);
                519 app.waitForState(job, JobState.RUNNING);
            testJobRebootOnLastRetryOnUnregistrationFailure()  (2 usages found)
                532 app.waitForState(job, JobState.RUNNING);
                545 app.waitForState(job, JobState.RUNNING);
            testCountersOnJobFinish()  (1 usage found)
                570 app.waitForState(job, JobState.SUCCEEDED);
            testContainerPassThrough()  (1 usage found)
                617 app.waitForState(job, JobState.SUCCEEDED);
        TestMRAppComponentDependencies  (1 usage found)
            testComponentStopOrder()  (1 usage found)
                47 app.waitForState(job, JobState.SUCCEEDED);
        TestMRClientService  (3 usages found)
            test()  (2 usages found)
                84 app.waitForState(job, JobState.RUNNING);
                186 app.waitForState(job, JobState.SUCCEEDED);
            testViewAclOnlyCannotModify()  (1 usage found)
                209 app.waitForState(job, JobState.RUNNING);
        TestRecovery  (38 usages found)
            testCrashed()  (3 usages found)
                146 app.waitForState(job, JobState.RUNNING);
                261 app.waitForState(job, JobState.RUNNING);
                297 app.waitForState(job, JobState.SUCCEEDED);
            testCrashOfMapsOnlyJob()  (3 usages found)
                359 app.waitForState(job, JobState.RUNNING);
                418 app.waitForState(job, JobState.RUNNING);
                448 app.waitForState(job, JobState.SUCCEEDED);
            testRecoverySuccessUsingCustomOutputCommitter()  (3 usages found)
                504 app.waitForState(job, JobState.RUNNING);
                564 app.waitForState(job, JobState.RUNNING);
                593 app.waitForState(job, JobState.SUCCEEDED);
            testRecoveryWithSpillEncryption()  (3 usages found)
                613 app.waitForState(jobAttempt1, JobState.RUNNING);
                639 app.waitForState(jobAttempt2, JobState.RUNNING);
                660 app.waitForState(jobAttempt2, JobState.SUCCEEDED);
            testRecoveryFailsUsingCustomOutputCommitter()  (3 usages found)
                683 app.waitForState(job, JobState.RUNNING);
                743 app.waitForState(job, JobState.RUNNING);
                786 app.waitForState(job, JobState.SUCCEEDED);
            testMultipleCrashes()  (4 usages found)
                803 app.waitForState(job, JobState.RUNNING);
                852 app.waitForState(job, JobState.RUNNING);
                895 app.waitForState(job, JobState.RUNNING);
                916 app.waitForState(job, JobState.SUCCEEDED);
            testOutputRecovery()  (3 usages found)
                931 app.waitForState(job, JobState.RUNNING);
                989 app.waitForState(job, JobState.RUNNING);
                1024 app.waitForState(job, JobState.SUCCEEDED);
            testPreviousJobOutputCleanedWhenNoRecovery()  (2 usages found)
                1042 app.waitForState(job, JobState.RUNNING);
                1052 app.waitForState(job, JobState.RUNNING);
            testPreviousJobIsNotCleanedWhenRecovery()  (2 usages found)
                1077 app.waitForState(job, JobState.RUNNING);
                1087 app.waitForState(job, JobState.RUNNING);
            testOutputRecoveryMapsOnly()  (3 usages found)
                1107 app.waitForState(job, JobState.RUNNING);
                1155 app.waitForState(job, JobState.RUNNING);
                1206 app.waitForState(job, JobState.SUCCEEDED);
            testRecoveryWithOldCommiter()  (3 usages found)
                1222 app.waitForState(job, JobState.RUNNING);
                1280 app.waitForState(job, JobState.RUNNING);
                1315 app.waitForState(job, JobState.SUCCEEDED);
            testSpeculative()  (3 usages found)
                1340 app.waitForState(job, JobState.RUNNING);
                1415 app.waitForState(job, JobState.RUNNING);
                1452 app.waitForState(job, JobState.SUCCEEDED);
            testRecoveryWithoutShuffleSecret()  (3 usages found)
                1493 app.waitForState(job, JobState.RUNNING);
                1540 app.waitForState(job, JobState.RUNNING);
                1586 app.waitForState(job, JobState.SUCCEEDED);
        TestStagingCleanup  (1 usage found)
            testStagingCleanupOrder()  (1 usage found)
                591 app.waitForState(job, JobState.SUCCEEDED);
    org.apache.hadoop.mapreduce.v2.app.job.impl  (12 usages found)
        TestMapReduceChildJVM  (7 usages found)
            testCommandLine()  (1 usage found)
                56 app.waitForState(job, JobState.SUCCEEDED);
            testReduceCommandLine(Configuration)  (1 usage found)
                112 app.waitForState(job, JobState.SUCCEEDED);
            testCommandLineWithLog4JConifg()  (1 usage found)
                161 app.waitForState(job, JobState.SUCCEEDED);
            testAutoHeapSize(int, int, String)  (1 usage found)
                230 app.waitForState(job, JobState.SUCCEEDED);
            testEnvironmentVariables()  (3 usages found)
                288 app.waitForState(job, JobState.SUCCEEDED);
                304 app.waitForState(job, JobState.SUCCEEDED);
                318 app.waitForState(job, JobState.SUCCEEDED);
        TestTaskAttempt  (5 usages found)
            verifyMillisCounters(Resource, int)  (2 usages found)
                356 app.waitForState(job, JobState.RUNNING);
                380 app.waitForState(job, JobState.SUCCEEDED);
            testMRAppHistory(MRApp)  (1 usage found)
                446 app.waitForState(job, JobState.FAILED);
            testTaskAttemptAssignedFailHistory(FailingAttemptsDuringAssignedMRApp)  (1 usage found)
                472 app.waitForState(job, JobState.FAILED);
            testTaskAttemptAssignedKilledHistory(FailingAttemptsDuringAssignedMRApp)  (1 usage found)
                482 app.waitForState(job, JobState.RUNNING);
    org.apache.hadoop.mapreduce.v2.app.launcher  (2 usages found)
        TestContainerLauncher  (2 usages found)
            testSlowNM()  (2 usages found)
                290 app.waitForState(job, JobState.RUNNING);
                307 app.waitForState(job, JobState.FAILED);
    org.apache.hadoop.mapreduce.v2.app.webapp  (4 usages found)
        TestAMWebApp  (4 usages found)
            testMRWebAppSSLDisabled()  (1 usage found)
                229 app.waitForState(job, JobState.SUCCEEDED);
            testMRWebAppSSLEnabled()  (1 usage found)
                286 app.waitForState(job, JobState.SUCCEEDED);
            testMRWebAppSSLEnabledWithClientAuth()  (1 usage found)
                357 app.waitForState(job, JobState.SUCCEEDED);
            testMRWebAppRedirection()  (1 usage found)
                411 app.waitForState(job, JobState.SUCCEEDED);
    org.apache.hadoop.mapreduce.v2.hs  (12 usages found)
        TestJobHistoryEvents  (3 usages found)
            testHistoryEvents()  (1 usage found)
                62 app.waitForState(job, JobState.SUCCEEDED);
            testEventsFlushOnStop()  (1 usage found)
                123 app.waitForState(job, JobState.SUCCEEDED);
            testAssignedQueue()  (1 usage found)
                171 app.waitForState(job, JobState.SUCCEEDED);
        TestJobHistoryParsing  (8 usages found)
            checkHistoryParsing(int, int, int)  (1 usage found)
                176 app.waitForState(job, JobState.SUCCEEDED);
            testHistoryParsingForFailedAttempts()  (1 usage found)
                416 app.waitForState(job, JobState.SUCCEEDED);
            testHistoryParsingForKilledAndFailedAttempts()  (1 usage found)
                492 app.waitForState(job, JobState.SUCCEEDED);
            testCountersForFailedTask()  (1 usage found)
                551 app.waitForState(job, JobState.FAILED);
            testDiagnosticsForKilledJob()  (1 usage found)
                614 app.waitForState(job, JobState.KILLED);
            testScanningOldDirs()  (1 usage found)
                674 app.waitForState(job, JobState.SUCCEEDED);
            testDeleteFileInfo()  (1 usage found)
                833 app.waitForState(job, JobState.SUCCEEDED);
            testJobHistoryMethods()  (1 usage found)
                888 app.waitForState(job, JobState.SUCCEEDED);
        TestJobHistoryServer  (1 usage found)
            testReports()  (1 usage found)
                106 app.waitForState(job, JobState.SUCCEEDED);

--------------------------------------------------------------------------------------

Method
    waitForState(STATE)
Found usages  (12 usages found)
    org.apache.hadoop.mapreduce.v2.app  (1 usage found)
        TestKill  (1 usage found)
            testKillJob()  (1 usage found)
                83 app.waitForState(Service.STATE.STOPPED);
    org.apache.hadoop.mapreduce.v2.hs  (11 usages found)
        TestJobHistoryEvents  (3 usages found)
            testHistoryEvents()  (1 usage found)
                65 app.waitForState(Service.STATE.STOPPED);
            testEventsFlushOnStop()  (1 usage found)
                126 app.waitForState(Service.STATE.STOPPED);
            testAssignedQueue()  (1 usage found)
                174 app.waitForState(Service.STATE.STOPPED);
        TestJobHistoryParsing  (8 usages found)
            checkHistoryParsing(int, int, int)  (1 usage found)
                179 app.waitForState(Service.STATE.STOPPED);
            testHistoryParsingForFailedAttempts()  (1 usage found)
                419 app.waitForState(Service.STATE.STOPPED);
            testHistoryParsingForKilledAndFailedAttempts()  (1 usage found)
                495 app.waitForState(Service.STATE.STOPPED);
            testCountersForFailedTask()  (1 usage found)
                554 app.waitForState(Service.STATE.STOPPED);
            testDiagnosticsForKilledJob()  (1 usage found)
                617 app.waitForState(Service.STATE.STOPPED);
            testScanningOldDirs()  (1 usage found)
                677 app.waitForState(Service.STATE.STOPPED);
            testDeleteFileInfo()  (1 usage found)
                836 app.waitForState(Service.STATE.STOPPED);
            testJobHistoryMethods()  (1 usage found)
                890 app.waitForState(Service.STATE.STOPPED);


--------------------------------------------------------------------------------------

Method
    waitForState(STATE)
Found usages  (12 usages found)
    org.apache.hadoop.mapreduce.v2.app  (1 usage found)
        TestKill  (1 usage found)
            testKillJob()  (1 usage found)
                83 app.waitForState(Service.STATE.STOPPED);
    org.apache.hadoop.mapreduce.v2.hs  (11 usages found)
        TestJobHistoryEvents  (3 usages found)
            testHistoryEvents()  (1 usage found)
                65 app.waitForState(Service.STATE.STOPPED);
            testEventsFlushOnStop()  (1 usage found)
                126 app.waitForState(Service.STATE.STOPPED);
            testAssignedQueue()  (1 usage found)
                174 app.waitForState(Service.STATE.STOPPED);
        TestJobHistoryParsing  (8 usages found)
            checkHistoryParsing(int, int, int)  (1 usage found)
                179 app.waitForState(Service.STATE.STOPPED);
            testHistoryParsingForFailedAttempts()  (1 usage found)
                419 app.waitForState(Service.STATE.STOPPED);
            testHistoryParsingForKilledAndFailedAttempts()  (1 usage found)
                495 app.waitForState(Service.STATE.STOPPED);
            testCountersForFailedTask()  (1 usage found)
                554 app.waitForState(Service.STATE.STOPPED);
            testDiagnosticsForKilledJob()  (1 usage found)
                617 app.waitForState(Service.STATE.STOPPED);
            testScanningOldDirs()  (1 usage found)
                677 app.waitForState(Service.STATE.STOPPED);
            testDeleteFileInfo()  (1 usage found)
                836 app.waitForState(Service.STATE.STOPPED);
            testJobHistoryMethods()  (1 usage found)
                890 app.waitForState(Service.STATE.STOPPED);

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

MAPREDUCE-7262-elapsedTimes.pdf
27/Jan/20 16:56
30 kB
Ahmed Hussein
MAPREDUCE-7262.001.patch
27/Jan/20 17:01
9 kB
Ahmed Hussein
MAPREDUCE-7262.002.patch
27/Jan/20 20:25
9 kB
Ahmed Hussein
MAPREDUCE-7262-branch-2.10.002.patch
27/Jan/20 23:28
9 kB
Ahmed Hussein

Activity

People

Assignee:: Ahmed Hussein

Reporter:: Ahmed Hussein

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 24/Jan/20 21:46

Updated:: 28/Jan/20 15:16

Resolved:: 27/Jan/20 22:44