Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

• Key: MAPREDUCE-382
• Type: Sub-task
• Status: Closed
• Resolution: Fixed
• Priority: Major
• Assignee: Devaraj Das
• Reporter: Devaraj Das

Description

Create a test that would inject random failures for tasks in large jobs and would also inject TaskTracker failures

Comments

[Devaraj Das]

Attaching a patch for review. I am still tuning some parameters in the tests but the patch can be reviewed as such.

[Arun C Murthy]

I only started looking through this, a nit: we shouldn't do 'ps | grep | ... ' etc. for suspend/resume. Rather we should use the pid file to get the daemon's pid.

[Devaraj Das]

Arun, the PID file is looked at if it exists. I have the 'ps | grep | etc' to take care of the case where the test is run using HOD (as is normally the case in Yahoo!). In that case, the PID file is not written. Also, note that stopping/resuming daemons through 'ps | grep | etc' will stop/resume only those processes that had been launched by the user running the test.

[Vinod K V]

In the minimum, we should do our best and ensure that we are sending signals to the right process. For this, we might want to grep process list for "java" AND the full class name, instead of just searching for the daemons' names. We are limiting to one user anyways, and further we are just stopping and continuing the process, and not really destroying them.

Gone throught the patch, a few code comments:

• runSleepJobTest, runRandomWriterTest and runSortTest can be refactored, they share much code.
• If we can somehow get TASKTRACKER_EXPIRY_INTERVAL from JT - either via a public API or may be via clusterStatus - it would make the tests better as compared to just relying on the user input.
• In KillTrackerThread.{startTaskTrackers|stopTaskTrackers}, the output of the shellCommand is currently discarded. That, along with the return code will give more information about the success of the signal sent.
• If configuration is not setup(mapred-site.xml), local jobrunner would be used and the test fails with little error reporting. I think we can check for jt configuration in the minimum.

[Devaraj Das]

Attached patch with the concerns addressed

[Vinod K V]

Few comments:

• The patch is breaking compilation because of the change in ClusterStatus constructor:
  • src/mapred/org/apache/hadoop/mapred/LocalJobRunner.java +389
  • src/test/org/apache/hadoop/mapred/TestJobQueueTaskScheduler.java:138
• When sleepJob(or rather examples) are not on the path, it fails but with the output as follows:

  JOB org.apache.hadoop.examples.SleepJob failed to run
              Waiting for the job org.apache.hadoop.examples.SleepJob to start

  We should avoid the last line, if we can.

• We can report progress of the jobs every once in a while when running the tests. Now it just stays dumb till the progress reaches the threshold values.
• I think writing statements to a LOG is better than printing on standard output.
• With a HOD allocation, the lost TaskTrackers simulating testcase fails even though keys are setup. This is because hadoop-daemons.sh tries on remote nodes to change to the non-existend directory HADOOP_HOME.

  exec "$bin/slaves.sh" --config $HADOOP_CONF_DIR cd "$HADOOP_HOME" \; "$bin/hadoop-daemon.sh" --config $HADOOP_CONF_DIR "$@"

  A simple solution would be to throw away all changes from hadoop-daemon.sh and hadoop-daemons.sh and simpley use slaves.sh as follows:

  HOSTLIST=conf/_reliability_test_slaves_file_ ./bin/slaves.sh ls

• the -ww flag to ps (ps auxw -ww) is not available on cygwin. It only modifies screen output and can be avoided. A side nit that I observed is that SIGCONT doesn't seem to work on cygwin. That would make the lost tasktracker simulation test completely useless on cygwin.
• The randomness of failures is pretty peculiar in the tests. Though it can be admitted that it can be changed later if need be.

[Vinod K V]

We can report progress of the jobs every once in a while when running the tests. Now it just stays dumb till the progress reaches the threshold values.

Sorry about that, log4j jar was missing on my classpath. It actually DOES print the job progress.

[Devaraj Das]

Attached is the updated patch.

[Vinod K V]

+1 for the patch.

[Devaraj Das]

I just committed this.

[Robert Chansler]

Edit release note for publication; tests not user-facing.