Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1000

JobHistory.initDone() should retain the try ... catch in the body

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    1. mapred-1000-v2.patch
      7 kB
      Jothi Padmanabhan
    2. mapred-1000.patch
      6 kB
      Jothi Padmanabhan

      Activity

      Hong Tang created issue -
      Hide
      Hong Tang added a comment -

      MAPREDUCE-157 changed JobHistory.initDone() and removed the try...catch clause of the body. The try...catch body is necessary because otherwise, if an IOE is thrown during the execution, JT would be aborted. I observed it when testing MAPREDUCE-728.

      Symptom:

      org.apache.hadoop.fs.ChecksumException: Checksum error: file:/Users/htang/Documents/Work/workspace/hadoop-mapreduce/build/hadoop-mapred-0.21.0-dev/logs/history/job_200904211745_0010_geek5 at 523264
              at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:221)
              at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:238)
              at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:190)
              at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158)
              at java.io.DataInputStream.read(DataInputStream.java:83)
              at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:72)
              at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:45)
              at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:97)
              at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:220)
              at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192)
              at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:143)
              at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:55)
              at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1203)
              at org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:338)
              at org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveOldFiles(JobHistory.java:372)
              at org.apache.hadoop.mapreduce.jobhistory.JobHistory.initDone(JobHistory.java:145)
              at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:3900)
              at org.apache.hadoop.mapred.SimulatorJobTracker.<init>(SimulatorJobTracker.java:80)
      

      The previous run of the JT was killed, which leaves the job history file mismatching with CRC checksum.

      The selected patch segment that shows the removal of the try...catch clause:

      Before MAPREDUCE-157

      -  static boolean initDone(JobConf conf, FileSystem fs){
      -    try {
      -      //if completed job history location is set, use that
      -      String doneLocation = conf.
      -                       get("mapred.job.tracker.history.completed.location");
      -      if (doneLocation != null) {
      -        DONE = fs.makeQualified(new Path(doneLocation));
      -        DONEDIR_FS = fs;
      -      } else {
      -        DONE = new Path(LOG_DIR, "done");
      -        DONEDIR_FS = LOGDIR_FS;
      -      }
      -
      -      //If not already present create the done folder with appropriate 
      -      //permission
      -      if (!DONEDIR_FS.exists(DONE)) {
      -        LOG.info("Creating DONE folder at "+ DONE);
      -        if (! DONEDIR_FS.mkdirs(DONE, 
      -            new FsPermission(HISTORY_DIR_PERMISSION))) {
      -          throw new IOException("Mkdirs failed to create " + DONE.toString());
      -        }
      -      }
      -
      -      fileManager.start();
      -      //move the log files remaining from last run to the DONE folder
      -      //suffix the file name based on Jobtracker identifier so that history
      -      //files with same job id don't get over written in case of recovery.
      -      FileStatus[] files = LOGDIR_FS.listStatus(new Path(LOG_DIR));
      -      String jtIdentifier = fileManager.jobTracker.getTrackerIdentifier();
      -      String fileSuffix = "." + jtIdentifier + OLD_SUFFIX;
      -      for (FileStatus fileStatus : files) {
      -        Path fromPath = fileStatus.getPath();
      -        if (fromPath.equals(DONE)) { //DONE can be a subfolder of log dir
      -          continue;
      -        }
      -        LOG.info("Moving log file from last run: " + fromPath);
      -        Path toPath = new Path(DONE, fromPath.getName() + fileSuffix);
      -        fileManager.moveToDoneNow(fromPath, toPath);
      -      }
      -    } catch(IOException e) {
      -        LOG.error("Failed to initialize JobHistory log file", e); 
      -        disableHistory = true;
      -    }
      -    return !(disableHistory);
      -  }
      

      After MAPREDUCE-157

      +  /** Initialize the done directory and start the history cleaner thread */
      +  public void initDone(JobConf conf, FileSystem fs) throws IOException {
      +    //if completed job history location is set, use that
      +    String doneLocation =
      +      conf.get("mapred.job.tracker.history.completed.location");
      +    if (doneLocation != null) {
      +      done = fs.makeQualified(new Path(doneLocation));
      +      doneDirFs = fs;
      +    } else {
      +      done = logDirFs.makeQualified(new Path(logDir, "done"));
      +      doneDirFs = logDirFs;
      +    }
      +
      +    //If not already present create the done folder with appropriate 
      +    //permission
      +    if (!doneDirFs.exists(done)) {
      +      LOG.info("Creating DONE folder at "+ done);
      +      if (! doneDirFs.mkdirs(done, 
      +          new FsPermission(HISTORY_DIR_PERMISSION))) {
      +        throw new IOException("Mkdirs failed to create " + done.toString());
      +      }
      +    }
      +    LOG.info("Inited the done directory to " + done.toString());
      +
      +    moveOldFiles();
      +    startFileMoverThreads();
      +
      +    // Start the History Cleaner Thread
      +    long maxAgeOfHistoryFiles = conf.getLong(
      +        "mapreduce.cluster.jobhistory.maxage", DEFAULT_HISTORY_MAX_AGE);
      +    historyCleanerThread = new HistoryCleaner(maxAgeOfHistoryFiles);
      +    historyCleanerThread.start();
      +  }
      
      Show
      Hong Tang added a comment - MAPREDUCE-157 changed JobHistory.initDone() and removed the try...catch clause of the body. The try...catch body is necessary because otherwise, if an IOE is thrown during the execution, JT would be aborted. I observed it when testing MAPREDUCE-728 . Symptom: org.apache.hadoop.fs.ChecksumException: Checksum error: file:/Users/htang/Documents/Work/workspace/hadoop-mapreduce/build/hadoop-mapred-0.21.0-dev/logs/history/job_200904211745_0010_geek5 at 523264 at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:221) at org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:238) at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:190) at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:158) at java.io.DataInputStream.read(DataInputStream.java:83) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:72) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:45) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:97) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:220) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:192) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:143) at org.apache.hadoop.fs.LocalFileSystem.copyFromLocalFile(LocalFileSystem.java:55) at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1203) at org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:338) at org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveOldFiles(JobHistory.java:372) at org.apache.hadoop.mapreduce.jobhistory.JobHistory.initDone(JobHistory.java:145) at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:3900) at org.apache.hadoop.mapred.SimulatorJobTracker.<init>(SimulatorJobTracker.java:80) The previous run of the JT was killed, which leaves the job history file mismatching with CRC checksum. The selected patch segment that shows the removal of the try...catch clause: Before MAPREDUCE-157 - static boolean initDone(JobConf conf, FileSystem fs){ - try { - //if completed job history location is set, use that - String doneLocation = conf. - get("mapred.job.tracker.history.completed.location"); - if (doneLocation != null) { - DONE = fs.makeQualified(new Path(doneLocation)); - DONEDIR_FS = fs; - } else { - DONE = new Path(LOG_DIR, "done"); - DONEDIR_FS = LOGDIR_FS; - } - - //If not already present create the done folder with appropriate - //permission - if (!DONEDIR_FS.exists(DONE)) { - LOG.info("Creating DONE folder at "+ DONE); - if (! DONEDIR_FS.mkdirs(DONE, - new FsPermission(HISTORY_DIR_PERMISSION))) { - throw new IOException("Mkdirs failed to create " + DONE.toString()); - } - } - - fileManager.start(); - //move the log files remaining from last run to the DONE folder - //suffix the file name based on Jobtracker identifier so that history - //files with same job id don't get over written in case of recovery. - FileStatus[] files = LOGDIR_FS.listStatus(new Path(LOG_DIR)); - String jtIdentifier = fileManager.jobTracker.getTrackerIdentifier(); - String fileSuffix = "." + jtIdentifier + OLD_SUFFIX; - for (FileStatus fileStatus : files) { - Path fromPath = fileStatus.getPath(); - if (fromPath.equals(DONE)) { //DONE can be a subfolder of log dir - continue; - } - LOG.info("Moving log file from last run: " + fromPath); - Path toPath = new Path(DONE, fromPath.getName() + fileSuffix); - fileManager.moveToDoneNow(fromPath, toPath); - } - } catch(IOException e) { - LOG.error("Failed to initialize JobHistory log file", e); - disableHistory = true; - } - return !(disableHistory); - } After MAPREDUCE-157 + /** Initialize the done directory and start the history cleaner thread */ + public void initDone(JobConf conf, FileSystem fs) throws IOException { + //if completed job history location is set, use that + String doneLocation = + conf.get("mapred.job.tracker.history.completed.location"); + if (doneLocation != null) { + done = fs.makeQualified(new Path(doneLocation)); + doneDirFs = fs; + } else { + done = logDirFs.makeQualified(new Path(logDir, "done")); + doneDirFs = logDirFs; + } + + //If not already present create the done folder with appropriate + //permission + if (!doneDirFs.exists(done)) { + LOG.info("Creating DONE folder at "+ done); + if (! doneDirFs.mkdirs(done, + new FsPermission(HISTORY_DIR_PERMISSION))) { + throw new IOException("Mkdirs failed to create " + done.toString()); + } + } + LOG.info("Inited the done directory to " + done.toString()); + + moveOldFiles(); + startFileMoverThreads(); + + // Start the History Cleaner Thread + long maxAgeOfHistoryFiles = conf.getLong( + "mapreduce.cluster.jobhistory.maxage", DEFAULT_HISTORY_MAX_AGE); + historyCleanerThread = new HistoryCleaner(maxAgeOfHistoryFiles); + historyCleanerThread.start(); + }
      Hide
      Jothi Padmanabhan added a comment -

      bq,. IOE is thrown during the execution, JT would be aborted

      Actually we intentionally did that – catch Exceptions during History Initialization and exit JT. Otherwise, these exceptions could go totally unnoticed and we might end up with no history file at all. Infact, this was one of Sharad's review comments here

      Show
      Jothi Padmanabhan added a comment - bq,. IOE is thrown during the execution, JT would be aborted Actually we intentionally did that – catch Exceptions during History Initialization and exit JT. Otherwise, these exceptions could go totally unnoticed and we might end up with no history file at all. Infact, this was one of Sharad's review comments here
      Hide
      Hong Tang added a comment -

      That is a fair argument. It seems that we could ignore some IOE in moveOldFiles() and ensure the end results of this call is that all files are gone (either moved, or removed).

      Show
      Hong Tang added a comment - That is a fair argument. It seems that we could ignore some IOE in moveOldFiles() and ensure the end results of this call is that all files are gone (either moved, or removed).
      Hide
      Chris Douglas added a comment -

      Since checksum exceptions should be reasonably common for partial history files, would it make sense to use RawLocalFileSystem as the src fs in moveOldFiles?

      Show
      Chris Douglas added a comment - Since checksum exceptions should be reasonably common for partial history files, would it make sense to use RawLocalFileSystem as the src fs in moveOldFiles?
      Hide
      Jothi Padmanabhan added a comment -

      It seems that we could ignore some IOE in moveOldFiles() and ensure the end results of this call is that all files are gone (either moved, or removed).

      This looks like a good idea

      Since checksum exceptions should be reasonably common for partial history files, would it make sense to use RawLocalFileSystem as the src fs in moveOldFiles?

      Actually, the job history log location is governed by the conf setting "hadoop.job.history.location" and this could possibly be on a FS other than LocalFS. So, I do not think we could use RawLocalFileSystem.

      Show
      Jothi Padmanabhan added a comment - It seems that we could ignore some IOE in moveOldFiles() and ensure the end results of this call is that all files are gone (either moved, or removed). This looks like a good idea Since checksum exceptions should be reasonably common for partial history files, would it make sense to use RawLocalFileSystem as the src fs in moveOldFiles? Actually, the job history log location is governed by the conf setting "hadoop.job.history.location" and this could possibly be on a FS other than LocalFS. So, I do not think we could use RawLocalFileSystem.
      Jothi Padmanabhan made changes -
      Field Original Value New Value
      Assignee Jothi Padmanabhan [ jothipn ]
      Hide
      Jothi Padmanabhan added a comment -

      Patch that tries to delete the history on exception while trying to move them to done folder.
      Also attached a test case that demonstrates the fix – it fails with trunk and passes with the patch

      Show
      Jothi Padmanabhan added a comment - Patch that tries to delete the history on exception while trying to move them to done folder. Also attached a test case that demonstrates the fix – it fails with trunk and passes with the patch
      Jothi Padmanabhan made changes -
      Attachment mapred-1000.patch [ 12420354 ]
      Jothi Padmanabhan made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Hong Tang added a comment -

      Patch looks good. One minor nit, pls use junit4 in the newly introduced unit test.

      Show
      Hong Tang added a comment - Patch looks good. One minor nit, pls use junit4 in the newly introduced unit test.
      Hide
      Sharad Agarwal added a comment -

      There is already a test case in TestJobHistory.testDoneFolderOnHDFS() which tests the moveToDone() code path on start up. Instead of adding a full new MiniMRCluster based testcase I think we can enhance a bit to test this. Also it would be good to assert that corrupt files got deleted from the current log directory and didn't move to done folder.

      Show
      Sharad Agarwal added a comment - There is already a test case in TestJobHistory.testDoneFolderOnHDFS() which tests the moveToDone() code path on start up. Instead of adding a full new MiniMRCluster based testcase I think we can enhance a bit to test this. Also it would be good to assert that corrupt files got deleted from the current log directory and didn't move to done folder.
      Hide
      Jothi Padmanabhan added a comment -

      Had an offline discussion with Sharad. We think we should actually delete only in the case of Checksum Exception. For other exceptions, we should just log and continue. For example, if the done directory does not have permissions for write, we might get an exception while trying to move and we do not want to delete in that case. Thoughts?

      Show
      Jothi Padmanabhan added a comment - Had an offline discussion with Sharad. We think we should actually delete only in the case of Checksum Exception. For other exceptions, we should just log and continue. For example, if the done directory does not have permissions for write, we might get an exception while trying to move and we do not want to delete in that case. Thoughts?
      Jothi Padmanabhan made changes -
      Status Patch Available [ 10002 ] Open [ 1 ]
      Jothi Padmanabhan made changes -
      Fix Version/s 0.21.0 [ 12314045 ]
      Affects Version/s 0.21.0 [ 12314045 ]
      Component/s jobtracker [ 12312907 ]
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12420354/mapred-1000.patch
      against trunk revision 817740.

      +1 @author. The patch does not contain any @author tags.

      +1 tests included. The patch appears to include 3 new or modified tests.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      -1 core tests. The patch failed core unit tests.

      -1 contrib tests. The patch failed contrib unit tests.

      Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/testReport/
      Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
      Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/artifact/trunk/build/test/checkstyle-errors.html
      Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420354/mapred-1000.patch against trunk revision 817740. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/125/console This message is automatically generated.
      Hide
      Jothi Padmanabhan added a comment -

      Patch that does the delete only on checksum exception.
      Also, moved the test case inside TestJobHistory so that job history test cases are consolidated in one place.

      Show
      Jothi Padmanabhan added a comment - Patch that does the delete only on checksum exception. Also, moved the test case inside TestJobHistory so that job history test cases are consolidated in one place.
      Jothi Padmanabhan made changes -
      Attachment mapred-1000-v2.patch [ 12420379 ]
      Jothi Padmanabhan made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch
      against trunk revision 817740.

      +1 @author. The patch does not contain any @author tags.

      +1 tests included. The patch appears to include 3 new or modified tests.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      -1 core tests. The patch failed core unit tests.

      -1 contrib tests. The patch failed contrib unit tests.

      Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/testReport/
      Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
      Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/artifact/trunk/build/test/checkstyle-errors.html
      Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch against trunk revision 817740. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/126/console This message is automatically generated.
      Jothi Padmanabhan made changes -
      Status Patch Available [ 10002 ] Open [ 1 ]
      Hide
      Jothi Padmanabhan added a comment -

      Trying Hudson again

      Show
      Jothi Padmanabhan added a comment - Trying Hudson again
      Jothi Padmanabhan made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch
      against trunk revision 818355.

      +1 @author. The patch does not contain any @author tags.

      +1 tests included. The patch appears to include 3 new or modified tests.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      -1 core tests. The patch failed core unit tests.

      -1 contrib tests. The patch failed contrib unit tests.

      Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/testReport/
      Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
      Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/artifact/trunk/build/test/checkstyle-errors.html
      Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch against trunk revision 818355. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/127/console This message is automatically generated.
      Hide
      Jothi Padmanabhan added a comment -

      Test failures are related to HADOOP-6281. WIll try Hudson once that is committed.

      Show
      Jothi Padmanabhan added a comment - Test failures are related to HADOOP-6281 . WIll try Hudson once that is committed.
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch
      against trunk revision 818355.

      +1 @author. The patch does not contain any @author tags.

      +1 tests included. The patch appears to include 3 new or modified tests.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      -1 findbugs. The patch appears to introduce new Findbugs warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      -1 core tests. The patch failed core unit tests.

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/testReport/
      Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
      Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/artifact/trunk/build/test/checkstyle-errors.html
      Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch against trunk revision 818355. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/128/console This message is automatically generated.
      Jothi Padmanabhan made changes -
      Status Patch Available [ 10002 ] Open [ 1 ]
      Hide
      Jothi Padmanabhan added a comment -

      The findbugs warning is spurious. The same patch gave +1 for findbugs in the previous run. TestCopyFiles failure is a known failure. Running by Hudson again, just in case

      Show
      Jothi Padmanabhan added a comment - The findbugs warning is spurious. The same patch gave +1 for findbugs in the previous run. TestCopyFiles failure is a known failure. Running by Hudson again, just in case
      Jothi Padmanabhan made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Hadoop QA added a comment -

      -1 overall. Here are the results of testing the latest attachment
      http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch
      against trunk revision 818674.

      +1 @author. The patch does not contain any @author tags.

      +1 tests included. The patch appears to include 3 new or modified tests.

      +1 javadoc. The javadoc tool did not generate any warning messages.

      +1 javac. The applied patch does not increase the total number of javac compiler warnings.

      +1 findbugs. The patch does not introduce any new Findbugs warnings.

      +1 release audit. The applied patch does not increase the total number of release audit warnings.

      -1 core tests. The patch failed core unit tests.

      +1 contrib tests. The patch passed contrib unit tests.

      Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/testReport/
      Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
      Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/artifact/trunk/build/test/checkstyle-errors.html
      Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/console

      This message is automatically generated.

      Show
      Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420379/mapred-1000-v2.patch against trunk revision 818674. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/58/console This message is automatically generated.
      Hide
      Jothi Padmanabhan added a comment -

      Failing test, TestCopyFiles is a known issue.

      Show
      Jothi Padmanabhan added a comment - Failing test, TestCopyFiles is a known issue.
      Hide
      Sharad Agarwal added a comment -

      +1
      I just committed this to 0.21 branch and trunk. Thanks Jothi.

      Show
      Sharad Agarwal added a comment - +1 I just committed this to 0.21 branch and trunk. Thanks Jothi.
      Sharad Agarwal made changes -
      Status Patch Available [ 10002 ] Resolved [ 5 ]
      Hadoop Flags [Reviewed]
      Resolution Fixed [ 1 ]
      Hide
      Hudson added a comment -

      Integrated in Hadoop-Mapreduce-trunk-Commit #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/65/)
      . Handle corrupt history files in JobHistory.initDone(). Contributed by Jothi Padmanabhan.

      Show
      Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #65 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/65/ ) . Handle corrupt history files in JobHistory.initDone(). Contributed by Jothi Padmanabhan.
      Tom White made changes -
      Status Resolved [ 5 ] Closed [ 6 ]

        People

        • Assignee:
          Jothi Padmanabhan
          Reporter:
          Hong Tang
        • Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development