[MAPREDUCE-5749] TestRMContainerAllocator#testReportedAppProgress Failed

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.5.0
• Target Version/s: 2.5.0
• Hadoop Flags: Reviewed
• Assignee: Jason Darrell Lowe
• Reporter: shenh062326
• Created: 10/Feb/14 05:31
• Updated: 10/Mar/15 04:30
• Resolved: 25/Apr/14 14:57

Description

When execute "mvn test -Dtest=TestRMContainerAllocator#testReportedAppProgress", It failed with message:

Caused by: java.io.FileNotFoundException: File /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001/job.xml does not exist

But in fact, the job.xml exits:

-rw-rw-r-- 1 yuling.sh yuling.sh 65791  2月 10 13:13 /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/yuling.sh/.staging/job_1392009213299_0001/job.xml

See the following code:

public Job submit(Configuration conf, boolean mapSpeculative,
      boolean reduceSpeculative) throws Exception {
    String user = conf.get(MRJobConfig.USER_NAME, UserGroupInformation
        .getCurrentUser().getShortUserName());
    conf.set(MRJobConfig.USER_NAME, user);
    conf.set(MRJobConfig.MR_AM_STAGING_DIR, testAbsPath.toString());
    conf.setBoolean(MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR, true);
    // TODO: fix the bug where the speculator gets events with
    // not-fully-constructed objects. For now, disable speculative exec
    conf.setBoolean(MRJobConfig.MAP_SPECULATIVE, mapSpeculative);
    conf.setBoolean(MRJobConfig.REDUCE_SPECULATIVE, reduceSpeculative);

    init(conf);
    start();
    DefaultMetricsSystem.shutdown();
    Job job = getContext().getAllJobs().values().iterator().next();
    if (assignedQueue != null) {
      job.setQueueName(assignedQueue);
    }

    // Write job.xml
    String jobFile = MRApps.getJobFile(conf, user,
        TypeConverter.fromYarn(job.getID()));
    LOG.info("Writing job conf to " + jobFile);
    new File(jobFile).getParentFile().mkdirs();
    conf.writeXml(new FileOutputStream(jobFile));

    return job;
  }

At first, user is "yuling.sh", but the UGI is setted to attemptId at "start();", after that, job.xml write to yuling.sh/.staging/job_1392009213299_0001/job.xml. But when the job is running, MRAppMaster can't find the job.xml at appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001.