Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2463

Job History files are not moving to done folder when job history location is hdfs location

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      If "mapreduce.jobtracker.jobhistory.location" is configured as HDFS location then either during initialization of Job Tracker (while moving old job history files) or after completion of the job, history files are not moving to done and giving following exception.

       
      2011-04-29 15:27:27,813 ERROR org.apache.hadoop.mapreduce.jobhistory.JobHistory: Unable to move history file to DONE folder.
      java.lang.IllegalArgumentException: Wrong FS: hdfs://10.18.52.146:9000/history/job_201104291518_0001_root, expected: file:///
      	at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:402)
      	at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:58)
      	at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:419)
      	at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:294)
      	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:215)
      	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1516)
      	at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1492)
      	at org.apache.hadoop.fs.FileSystem.moveFromLocalFile(FileSystem.java:1482)
      	at org.apache.hadoop.mapreduce.jobhistory.JobHistory.moveToDoneNow(JobHistory.java:348)
      	at org.apache.hadoop.mapreduce.jobhistory.JobHistory.access$200(JobHistory.java:61)
      	at org.apache.hadoop.mapreduce.jobhistory.JobHistory$1.run(JobHistory.java:439)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:619)
      
      
      1. MAPREDUCE-2463.patch
        3 kB
        Devaraj K
      2. MAPREDUCE-2463-1.patch
        4 kB
        Devaraj K
      3. MAPREDUCE-2463-2.patch
        4 kB
        Devaraj K

        Issue Links

          Activity

          Hide
          Philip Zeyliger added a comment -

          I recall MAPREDUCE-2351 changing how this code path worked. Don't entirely remember the details any more...

          Show
          Philip Zeyliger added a comment - I recall MAPREDUCE-2351 changing how this code path worked. Don't entirely remember the details any more...
          Hide
          Devaraj K added a comment -

          As per MAPREDUCE-2351, if we give any arbitrary file system URI for "mapred.job.tracker.history.completed.location" is working fine. But if we give other than local file system URI for "mapreduce.jobtracker.jobhistory.location" is failing.

          JobHistory.java
            private void moveToDoneNow(Path fromPath, Path toPath) throws IOException {
              //check if path exists, in case of retries it may not exist
              if (logDirFs.exists(fromPath)) {
                LOG.info("Moving " + fromPath.toString() + " to " +
                    toPath.toString());
                doneDirFs.moveFromLocalFile(fromPath, toPath);
                doneDirFs.setPermission(toPath,
                    new FsPermission(JobHistory.HISTORY_FILE_PERMISSION));
              }
            }
          

          In the above code, doneDirFs.moveFromLocalFile(fromPath, toPath); is trying to move from local file system even if we give the history location as hdfs location and it is failing with the above exception.

          Show
          Devaraj K added a comment - As per MAPREDUCE-2351 , if we give any arbitrary file system URI for "mapred.job.tracker.history.completed.location" is working fine. But if we give other than local file system URI for "mapreduce.jobtracker.jobhistory.location" is failing. JobHistory.java private void moveToDoneNow(Path fromPath, Path toPath) throws IOException { //check if path exists, in case of retries it may not exist if (logDirFs.exists(fromPath)) { LOG.info( "Moving " + fromPath.toString() + " to " + toPath.toString()); doneDirFs.moveFromLocalFile(fromPath, toPath); doneDirFs.setPermission(toPath, new FsPermission(JobHistory.HISTORY_FILE_PERMISSION)); } } In the above code, doneDirFs.moveFromLocalFile(fromPath, toPath); is trying to move from local file system even if we give the history location as hdfs location and it is failing with the above exception.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479082/MAPREDUCE-2463.patch
          against trunk revision 1102515.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these core unit tests:
          org.apache.hadoop.mapred.TestJobHistory

          -1 contrib tests. The patch failed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/242//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/242//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/242//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479082/MAPREDUCE-2463.patch against trunk revision 1102515. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapred.TestJobHistory -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/242//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/242//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/242//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          Looks like the unit test is failing

          Show
          Todd Lipcon added a comment - Looks like the unit test is failing
          Hide
          Devaraj K added a comment -

          Corrected the testcase and updated the patch.

          Show
          Devaraj K added a comment - Corrected the testcase and updated the patch.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479603/MAPREDUCE-2463-1.patch
          against trunk revision 1104687.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/263//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/263//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/263//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479603/MAPREDUCE-2463-1.patch against trunk revision 1104687. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/263//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/263//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/263//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          A few nits on the test cases:

          • can you please make the constants in TestJobHistory capitalized names, eg LOG_DIR and BUILD_DIR instead of hLogDir and buildDir?
          • the constant hLogDir is only used to constract the variable "logDir" twice - why not just make the constant like:
            private static final String LOCAL_LOG_DIR_URI = "file:///" + new File(System.getProperty("hadoop.log.dir")).getAbsolutePath() + File.separator + "history";
            

          If you could also add a brief javadoc for each of the test cases explaining what it's testing, that would be nice (for example "tests the case where the log directory is on local disk, the done folder is on HDFS, and the default FS is local"

          Show
          Todd Lipcon added a comment - A few nits on the test cases: can you please make the constants in TestJobHistory capitalized names, eg LOG_DIR and BUILD_DIR instead of hLogDir and buildDir? the constant hLogDir is only used to constract the variable "logDir" twice - why not just make the constant like: private static final String LOCAL_LOG_DIR_URI = "file: ///" + new File( System .getProperty( "hadoop.log.dir" )).getAbsolutePath() + File.separator + "history" ; If you could also add a brief javadoc for each of the test cases explaining what it's testing, that would be nice (for example "tests the case where the log directory is on local disk, the done folder is on HDFS, and the default FS is local"
          Hide
          Devaraj K added a comment -

          Thanks Todd for reviewing.

          Provided patch by addressing the above comments.

          Show
          Devaraj K added a comment - Thanks Todd for reviewing. Provided patch by addressing the above comments.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12479912/MAPREDUCE-2463-2.patch
          against trunk revision 1125428.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/285//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/285//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/285//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479912/MAPREDUCE-2463-2.patch against trunk revision 1125428. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/285//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/285//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/285//console This message is automatically generated.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          I have committed this. Thanks, Devaraj!

          Show
          Tsz Wo Nicholas Sze added a comment - I have committed this. Thanks, Devaraj!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #760 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/760/)
          MAPREDUCE-2463. Job history files are not moved to done folder when job history location is hdfs. Contributed by Devaraj K

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151722
          Files :

          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java
          • /hadoop/common/trunk/mapreduce/CHANGES.txt
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobHistory.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #760 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/760/ ) MAPREDUCE-2463 . Job history files are not moved to done folder when job history location is hdfs. Contributed by Devaraj K szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151722 Files : /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobHistory.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #751 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/751/)
          MAPREDUCE-2463. Job history files are not moved to done folder when job history location is hdfs. Contributed by Devaraj K

          szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151722
          Files :

          • /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java
          • /hadoop/common/trunk/mapreduce/CHANGES.txt
          • /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobHistory.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #751 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/751/ ) MAPREDUCE-2463 . Job history files are not moved to done folder when job history location is hdfs. Contributed by Devaraj K szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151722 Files : /hadoop/common/trunk/mapreduce/src/java/org/apache/hadoop/mapreduce/jobhistory/JobHistory.java /hadoop/common/trunk/mapreduce/CHANGES.txt /hadoop/common/trunk/mapreduce/src/test/mapred/org/apache/hadoop/mapred/TestJobHistory.java

            People

            • Assignee:
              Devaraj K
              Reporter:
              Devaraj K
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development