Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5749

TestRMContainerAllocator#testReportedAppProgress Failed

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.5.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When execute "mvn test -Dtest=TestRMContainerAllocator#testReportedAppProgress", It failed with message:

      Caused by: java.io.FileNotFoundException: File /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001/job.xml does not exist
      

      But in fact, the job.xml exits:

      -rw-rw-r-- 1 yuling.sh yuling.sh 65791  2月 10 13:13 /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/yuling.sh/.staging/job_1392009213299_0001/job.xml
      

      See the following code:

      public Job submit(Configuration conf, boolean mapSpeculative,
            boolean reduceSpeculative) throws Exception {
          String user = conf.get(MRJobConfig.USER_NAME, UserGroupInformation
              .getCurrentUser().getShortUserName());
          conf.set(MRJobConfig.USER_NAME, user);
          conf.set(MRJobConfig.MR_AM_STAGING_DIR, testAbsPath.toString());
          conf.setBoolean(MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR, true);
          // TODO: fix the bug where the speculator gets events with
          // not-fully-constructed objects. For now, disable speculative exec
          conf.setBoolean(MRJobConfig.MAP_SPECULATIVE, mapSpeculative);
          conf.setBoolean(MRJobConfig.REDUCE_SPECULATIVE, reduceSpeculative);
      
          init(conf);
          start();
          DefaultMetricsSystem.shutdown();
          Job job = getContext().getAllJobs().values().iterator().next();
          if (assignedQueue != null) {
            job.setQueueName(assignedQueue);
          }
      
          // Write job.xml
          String jobFile = MRApps.getJobFile(conf, user,
              TypeConverter.fromYarn(job.getID()));
          LOG.info("Writing job conf to " + jobFile);
          new File(jobFile).getParentFile().mkdirs();
          conf.writeXml(new FileOutputStream(jobFile));
      
          return job;
        }
      

      At first, user is "yuling.sh", but the UGI is setted to attemptId at "start();", after that, job.xml write to yuling.sh/.staging/job_1392009213299_0001/job.xml. But when the job is running, MRAppMaster can't find the job.xml at appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001.

        Attachments

        1. MAPREDUCE-5749.patch
          1 kB
          Hong Shen
        2. MAPREDUCE-5749v2.patch
          2 kB
          Jason Lowe
        3. TestRMContainerAllocator_failure.txt
          435 kB
          Kihwal Lee

          Activity

            People

            • Assignee:
              jlowe Jason Lowe
              Reporter:
              shenhong Hong Shen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: