Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      This feature provides facility to store job history on DFS. Now cluster admin can provide either localFS location or DFS location using configuration property "mapred.job.history.location" to store job histroy. History will be logged in user specified location also. User can specify history location using configuration property "mapred.job.history.user.location" .
      The classes org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndex and org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndexParseListener, and public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseMasterIndex are not available.
      The signature of public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseJobTasks(File jobHistoryFile, JobHistory.JobInfo job) is changed to DefaultJobHistoryParser.parseJobTasks(String jobHistoryFile, JobHistory.JobInfo job, FileSystem fs).
      The signature of public method org.apache.hadoop.mapred.JobHistory.parseHistory(File path, Listener l) is changed to JobHistory.parseHistoryFromFS(String path, Listener l, FileSystem fs)
      Show
      This feature provides facility to store job history on DFS. Now cluster admin can provide either localFS location or DFS location using configuration property "mapred.job.history.location" to store job histroy. History will be logged in user specified location also. User can specify history location using configuration property "mapred.job.history.user.location" . The classes org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndex and org.apache.hadoop.mapred.DefaultJobHistoryParser.MasterIndexParseListener, and public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseMasterIndex are not available. The signature of public method org.apache.hadoop.mapred.DefaultJobHistoryParser.parseJobTasks(File jobHistoryFile, JobHistory.JobInfo job) is changed to DefaultJobHistoryParser.parseJobTasks(String jobHistoryFile, JobHistory.JobInfo job, FileSystem fs). The signature of public method org.apache.hadoop.mapred.JobHistory.parseHistory(File path, Listener l) is changed to JobHistory.parseHistoryFromFS(String path, Listener l, FileSystem fs)

      Description

      This issue addresses the following items :

      1. Check for accuracy of job tracker history logs.

      2. After completion of the job, copy the JobHistory.log(Master index file) and the job history files to the DFS.

      3. User can load the history with commands
      bin/hadoop job -history <directory>
      or
      bin/hadoop job -history <jobid>
      This will start a stand-alone jetty and load jsps

      1. patch-2178.txt
        110 kB
        Amareshwari Sriramadasu
      2. patch-2178.2.txt
        110 kB
        Amareshwari Sriramadasu
      3. patch-2178.txt
        108 kB
        Amareshwari Sriramadasu
      4. patch-2178.txt
        108 kB
        Amareshwari Sriramadasu
      5. patch-2178.txt
        107 kB
        Amareshwari Sriramadasu
      6. patch-2178.txt
        105 kB
        Amareshwari Sriramadasu
      7. patch-2178.txt
        104 kB
        Amareshwari Sriramadasu

        Issue Links

          Activity

          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #406 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/406/ )
          Hide
          Devaraj Das added a comment -

          I just committed this. Thanks, Amareshwari!

          Show
          Devaraj Das added a comment - I just committed this. Thanks, Amareshwari!
          Hide
          Amareshwari Sriramadasu added a comment -

          Release audit warnings are due to jsp files.

          Show
          Amareshwari Sriramadasu added a comment - Release audit warnings are due to jsp files.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12375810/patch-2178.txt
          against trunk revision 619744.

          @author +1. The patch does not contain any @author tags.

          tests included +1. The patch appears to include 19 new or modified tests.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit -1. The applied patch generated 180 release audit warnings (more than the trunk's current 176 warnings).

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1812/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1812/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1812/artifact/trunk/build/test/checkstyle-errors.html
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1812/artifact/trunk/current/releaseAuditDiffWarnings.txt
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1812/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12375810/patch-2178.txt against trunk revision 619744. @author +1. The patch does not contain any @author tags. tests included +1. The patch appears to include 19 new or modified tests. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit -1. The applied patch generated 180 release audit warnings (more than the trunk's current 176 warnings). findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1812/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1812/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1812/artifact/trunk/build/test/checkstyle-errors.html Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1812/artifact/trunk/current/releaseAuditDiffWarnings.txt Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1812/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Regenerated patch with trunk

          Show
          Amareshwari Sriramadasu added a comment - Regenerated patch with trunk
          Hide
          Devaraj Das added a comment -

          Sorry, this patch doesn't apply cleanly anymore. Please regenerate the patch.

          Show
          Devaraj Das added a comment - Sorry, this patch doesn't apply cleanly anymore. Please regenerate the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12374749/patch-2178.2.txt
          against trunk revision 616796.

          @author +1. The patch does not contain any @author tags.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new javac compiler warnings.

          release audit -1. The applied patch generated 179 release audit warnings (more than the trunk's current 175 warnings).

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1740/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1740/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1740/artifact/trunk/build/test/checkstyle-errors.html
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1740/artifact/trunk/current/releaseAuditDiffWarnings.txt
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1740/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12374749/patch-2178.2.txt against trunk revision 616796. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new javac compiler warnings. release audit -1. The applied patch generated 179 release audit warnings (more than the trunk's current 175 warnings). findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1740/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1740/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1740/artifact/trunk/build/test/checkstyle-errors.html Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1740/artifact/trunk/current/releaseAuditDiffWarnings.txt Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1740/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Submiting patch again. Now all the tests pass after fixing HADOOP-2754. I added license header to OutputLogFilter. Replaced calls to deprecated listpaths.

          Show
          Amareshwari Sriramadasu added a comment - Submiting patch again. Now all the tests pass after fixing HADOOP-2754 . I added license header to OutputLogFilter. Replaced calls to deprecated listpaths.
          Hide
          Amareshwari Sriramadasu added a comment -

          canceling patch to fix hudson warnings

          Show
          Amareshwari Sriramadasu added a comment - canceling patch to fix hudson warnings
          Hide
          Nigel Daley added a comment -

          The only release audit warning that matters is that src/java/org/apache/hadoop/mapred/OutputLogFilter.java is missing a license header. Also, did you look at the new javac warning?

          Show
          Nigel Daley added a comment - The only release audit warning that matters is that src/java/org/apache/hadoop/mapred/OutputLogFilter.java is missing a license header. Also, did you look at the new javac warning?
          Hide
          Amareshwari Sriramadasu added a comment - - edited

          The test failure is not related to the patch.

          Show
          Amareshwari Sriramadasu added a comment - - edited The test failure is not related to the patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12374003/patch-2178.txt
          against trunk revision 615723.

          @author +1. The patch does not contain any @author tags.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac -1. The applied patch generated 608 javac compiler warnings (more than the trunk's current 607 warnings).

          release audit -1. The applied patch generated 212 release audit warnings (more than the trunk's current 207 warnings).

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests -1. The patch failed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1710/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1710/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1710/artifact/trunk/build/test/checkstyle-errors.html
          Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1710/artifact/trunk/current/releaseAuditDiffWarnings.txt
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1710/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12374003/patch-2178.txt against trunk revision 615723. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac -1. The applied patch generated 608 javac compiler warnings (more than the trunk's current 607 warnings). release audit -1. The applied patch generated 212 release audit warnings (more than the trunk's current 207 warnings). findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1710/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1710/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1710/artifact/trunk/build/test/checkstyle-errors.html Release audit warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1710/artifact/trunk/current/releaseAuditDiffWarnings.txt Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1710/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          trying hudson

          Show
          Amareshwari Sriramadasu added a comment - trying hudson
          Hide
          Nigel Daley added a comment -

          Since this is a 0.17 fix, I'm marking this "open" to get it out of our long patch build queue on Hudson. Please mark it "patch available" once we branch 0.16.

          Show
          Nigel Daley added a comment - Since this is a 0.17 fix, I'm marking this "open" to get it out of our long patch build queue on Hudson. Please mark it "patch available" once we branch 0.16.
          Hide
          Amareshwari Sriramadasu added a comment -

          Sorry again, userLogDir == "none" should use equals method.

          Show
          Amareshwari Sriramadasu added a comment - Sorry again, userLogDir == "none" should use equals method.
          Hide
          Nigel Daley added a comment -

          Deferring to 0.17

          Show
          Nigel Daley added a comment - Deferring to 0.17
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12373921/patch-2178.txt
          against trunk revision 614721.

          @author +1. The patch does not contain any @author tags.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new compiler warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1665/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1665/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1665/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1665/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12373921/patch-2178.txt against trunk revision 614721. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1665/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1665/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1665/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/1665/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Sorry, OutputLogFilter was not added to svn

          Show
          Amareshwari Sriramadasu added a comment - Sorry, OutputLogFilter was not added to svn
          Hide
          Amareshwari Sriramadasu added a comment -

          Submiting patch in sync with trunk.

          Show
          Amareshwari Sriramadasu added a comment - Submiting patch in sync with trunk.
          Hide
          Amareshwari Sriramadasu added a comment -

          patch not in sync with trunk.

          Show
          Amareshwari Sriramadasu added a comment - patch not in sync with trunk.
          Hide
          Amareshwari Sriramadasu added a comment -

          Submiting patch with Devaraj's comments incorporated.

          Along with the comments, This patch addresses couple of things:
          1. User can stop logging to output directory by giving value "none" to config item "hadoop.job.history.user.location".
          2. Adds a class OutputLogFilter to mapred package, the path filter, which filters the paths containing logs.

          Show
          Amareshwari Sriramadasu added a comment - Submiting patch with Devaraj's comments incorporated. Along with the comments, This patch addresses couple of things: 1. User can stop logging to output directory by giving value "none" to config item "hadoop.job.history.user.location". 2. Adds a class OutputLogFilter to mapred package, the path filter, which filters the paths containing logs.
          Hide
          Devaraj Das added a comment -

          Some comments (sorry for being late on this)
          1) Some of the configurable items are not listed in hadoop-default.xml. For example - mapred.job.history.http.bindAddress
          2) mapred.job.history.viewer seems redundant
          3) The documentation should call out those cases, where history files are getting created in the output directory, explicitly. Something along the lines of having to write a filter for listing the output directory esp for cases where the output dir would be consumed by a subsequent MR job. I think it makes sense to have the filter be prepackaged as part of hadoop. That way every user doesn't have to implement the same thing.
          4) The documentation should mention that the user should start a browser and connect to the host:port that he gets when he starts the viewer.

          Show
          Devaraj Das added a comment - Some comments (sorry for being late on this) 1) Some of the configurable items are not listed in hadoop-default.xml. For example - mapred.job.history.http.bindAddress 2) mapred.job.history.viewer seems redundant 3) The documentation should call out those cases, where history files are getting created in the output directory, explicitly. Something along the lines of having to write a filter for listing the output directory esp for cases where the output dir would be consumed by a subsequent MR job. I think it makes sense to have the filter be prepackaged as part of hadoop. That way every user doesn't have to implement the same thing. 4) The documentation should mention that the user should start a browser and connect to the host:port that he gets when he starts the viewer.
          Hide
          dhruba borthakur added a comment -

          This is currently being investigated in HADOOP-2680.

          Show
          dhruba borthakur added a comment - This is currently being investigated in HADOOP-2680 .
          Hide
          Amareshwari Sriramadasu added a comment -

          org.apache.hadoop.dfs.TestCrcCorruption.testCrcCorruption failed because datanodes are bad.

          Show
          Amareshwari Sriramadasu added a comment - org.apache.hadoop.dfs.TestCrcCorruption.testCrcCorruption failed because datanodes are bad.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12373741/patch-2178.txt
          against trunk revision r614192.

          @author +1. The patch does not contain any @author tags.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new compiler warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests -1. The patch failed core unit tests.

          contrib tests -1. The patch failed contrib unit tests.

          Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1676/testReport/
          Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1676/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1676/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1676/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12373741/patch-2178.txt against trunk revision r614192. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1676/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1676/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1676/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1676/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Submiting patch with documentation added to cluster_setup and mapred_tutorial. There is no change in the code.

          Show
          Amareshwari Sriramadasu added a comment - Submiting patch with documentation added to cluster_setup and mapred_tutorial. There is no change in the code.
          Hide
          Amareshwari Sriramadasu added a comment -

          Adding documentation

          Show
          Amareshwari Sriramadasu added a comment - Adding documentation
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12373591/patch-2178.txt
          against trunk revision r613499.

          @author +1. The patch does not contain any @author tags.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new compiler warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests +1. The patch passed contrib unit tests.

          Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1663/testReport/
          Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1663/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1663/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1663/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12373591/patch-2178.txt against trunk revision r613499. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests +1. The patch passed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1663/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1663/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1663/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1663/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          All tests passed on my machine with this patch

          Show
          Amareshwari Sriramadasu added a comment - All tests passed on my machine with this patch
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12373528/patch-2178.txt
          against trunk revision r613115.

          @author +1. The patch does not contain any @author tags.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new compiler warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests +1. The patch passed core unit tests.

          contrib tests -1. The patch failed contrib unit tests.

          Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1645/testReport/
          Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1645/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1645/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1645/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12373528/patch-2178.txt against trunk revision r613115. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests +1. The patch passed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1645/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1645/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1645/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1645/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          Submiting again with TestPipes fix

          Show
          Amareshwari Sriramadasu added a comment - Submiting again with TestPipes fix
          Hide
          Amareshwari Sriramadasu added a comment -

          TestPipes needs a pathFilter

          Show
          Amareshwari Sriramadasu added a comment - TestPipes needs a pathFilter
          Hide
          Amareshwari Sriramadasu added a comment -

          Submiting patch with comments incorporated and tested.

          Show
          Amareshwari Sriramadasu added a comment - Submiting patch with comments incorporated and tested.
          Hide
          Amareshwari Sriramadasu added a comment -

          Canceling patch for review comments from Devaraj

          Show
          Amareshwari Sriramadasu added a comment - Canceling patch for review comments from Devaraj
          Hide
          Amareshwari Sriramadasu added a comment -

          Submiting again for ant tests.

          Show
          Amareshwari Sriramadasu added a comment - Submiting again for ant tests.
          Hide
          Amareshwari Sriramadasu added a comment -

          looking into test failures

          Show
          Amareshwari Sriramadasu added a comment - looking into test failures
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12373372/patch-2178.txt
          against trunk revision r612995.

          @author +1. The patch does not contain any @author tags.

          javadoc +1. The javadoc tool did not generate any warning messages.

          javac +1. The applied patch does not generate any new compiler warnings.

          findbugs +1. The patch does not introduce any new Findbugs warnings.

          core tests -1. The patch failed core unit tests.

          contrib tests -1. The patch failed contrib unit tests.

          Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1630/testReport/
          Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1630/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1630/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1630/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12373372/patch-2178.txt against trunk revision r612995. @author +1. The patch does not contain any @author tags. javadoc +1. The javadoc tool did not generate any warning messages. javac +1. The applied patch does not generate any new compiler warnings. findbugs +1. The patch does not introduce any new Findbugs warnings. core tests -1. The patch failed core unit tests. contrib tests -1. The patch failed contrib unit tests. Test results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1630/testReport/ Findbugs warnings: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1630/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1630/artifact/trunk/build/test/checkstyle-errors.html Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/1630/console This message is automatically generated.
          Hide
          Amareshwari Sriramadasu added a comment -

          try hudson with the correct patch.

          Show
          Amareshwari Sriramadasu added a comment - try hudson with the correct patch.
          Hide
          Amareshwari Sriramadasu added a comment -

          This patch logs history in hadoop.job.history.location whose default value is local file system.
          In addition to that it logs at hadoop.job.history.user.location whose default value isuser output directory.

          And now the history jsp files are in seperate webapps i.e src/webapps/history, ratherthan in src/webapps/job .
          We start a statushttpserver from job tracker with hadoop.job.history.location as history log directory and view history files. This page is linked from job tracker.
          User can view history files in output directory by giving the following command
          bin/hadoop job -history <outputdir>
          This will start a jetty on that machine and user can view the jsps of the history directory he has given.

          Now we have centralised location of history as "hadoop.job.history.location" .

          Show
          Amareshwari Sriramadasu added a comment - This patch logs history in hadoop.job.history.location whose default value is local file system. In addition to that it logs at hadoop.job.history.user.location whose default value isuser output directory. And now the history jsp files are in seperate webapps i.e src/webapps/history, ratherthan in src/webapps/job . We start a statushttpserver from job tracker with hadoop.job.history.location as history log directory and view history files. This page is linked from job tracker. User can view history files in output directory by giving the following command bin/hadoop job -history <outputdir> This will start a jetty on that machine and user can view the jsps of the history directory he has given. Now we have centralised location of history as "hadoop.job.history.location" .
          Hide
          Amareshwari Sriramadasu added a comment -

          In the case of HOD deployed JTs, I think we can then either set hadoop.job.history.location to NULL or to a HOD specified output directory, probably on HDFS or another shared FS. This would be useful if the user is running a lot jobs through a single HOD instance. NULL is probably a fine default.

          In case of HOD, hadoop.job.history.location default value can be local file system, sothat HOD can collect logs at the shutdown of cluster as is today.

          Show
          Amareshwari Sriramadasu added a comment - In the case of HOD deployed JTs, I think we can then either set hadoop.job.history.location to NULL or to a HOD specified output directory, probably on HDFS or another shared FS. This would be useful if the user is running a lot jobs through a single HOD instance. NULL is probably a fine default. In case of HOD, hadoop.job.history.location default value can be local file system, sothat HOD can collect logs at the shutdown of cluster as is today.
          Hide
          eric baldeschwieler added a comment -

          The user vs admin dimension is not really the same as the static vs HOD dimension.

          Even in a static JT, the job history is probably usefully part of the output directory (for a user). I think this should just be part of the output API, no matter how the cluster is configured. This will be much easier to document and use.

          Then we could handle static job trackers as Amareshwari describes. In the case of HOD deployed JTs, I think we can then either set hadoop.job.history.location to NULL or to a HOD specified output directory, probably on HDFS or another shared FS. This would be useful if the user is running a lot jobs through a single HOD instance. NULL is probably a fine default.

          I agree with runping that we need to define an API for collecting central stats from HOD deployed JTs. I think a configured URL is ok as an API, but we need to be clear that this output will be for central collection, not user diagnostics and as such the layout should be optimized to simplify that (probably time sorted, not user sorted for example). Ideally this could be a single file per JT instance.

          Show
          eric baldeschwieler added a comment - The user vs admin dimension is not really the same as the static vs HOD dimension. Even in a static JT, the job history is probably usefully part of the output directory (for a user). I think this should just be part of the output API, no matter how the cluster is configured. This will be much easier to document and use. Then we could handle static job trackers as Amareshwari describes. In the case of HOD deployed JTs, I think we can then either set hadoop.job.history.location to NULL or to a HOD specified output directory, probably on HDFS or another shared FS. This would be useful if the user is running a lot jobs through a single HOD instance. NULL is probably a fine default. — I agree with runping that we need to define an API for collecting central stats from HOD deployed JTs. I think a configured URL is ok as an API, but we need to be clear that this output will be for central collection, not user diagnostics and as such the layout should be optimized to simplify that (probably time sorted, not user sorted for example). Ideally this could be a single file per JT instance.
          Hide
          Runping Qi added a comment -

          Even with hod JT, we still need to address case 3. That is,
          we need a central place to store the job history files so that they can be analyzed offline.
          It can be the same place in the local file system as it is now, or some common directory in DFS.
          This is in addition to the one in the output directory.

          Show
          Runping Qi added a comment - Even with hod JT, we still need to address case 3. That is, we need a central place to store the job history files so that they can be analyzed offline. It can be the same place in the local file system as it is now, or some common directory in DFS. This is in addition to the one in the output directory.
          Hide
          Amareshwari Sriramadasu added a comment -

          To address use cases 1 and 2 suggested by Eric, I propose the following approach.

          If the job tracker is static, we will store history logs in a location specified by hadoop.job.history.location, by default it is local file system.
          If the job tracker is not static (like HOD JT) we will store log files in user specified location, by default it is job output directory.

          We will not have index file any more, because appending becomes an issue in DFS. And we dont need one in case of non-static JT.
          For static JT, We can do listing of files in the log directory to show the first page.

          Show
          Amareshwari Sriramadasu added a comment - To address use cases 1 and 2 suggested by Eric, I propose the following approach. If the job tracker is static, we will store history logs in a location specified by hadoop.job.history.location, by default it is local file system. If the job tracker is not static (like HOD JT) we will store log files in user specified location, by default it is job output directory. We will not have index file any more, because appending becomes an issue in DFS. And we dont need one in case of non-static JT. For static JT, We can do listing of files in the log directory to show the first page.
          Hide
          Runping Qi added a comment -

          +1.

          This is much clearer.

          Need coordination in implementing this and HADOOP-1876.

          Show
          Runping Qi added a comment - +1. This is much clearer. Need coordination in implementing this and HADOOP-1876 .
          Hide
          Devaraj Das added a comment -

          Makes sense i think. Keeps things simple..

          Show
          Devaraj Das added a comment - Makes sense i think. Keeps things simple..
          Hide
          eric baldeschwieler added a comment -

          I don't think it makes sense to address HADOOP-1876 by storing history on HDFS by default.

          I think it is better to keep the server logs on the local machine by default and allow a configuration to store them in a well specified place on HDFS. This will keep a HOD user from creating leavings all over the HDFS, but allow for alternate uses like HADOOP-1876.

          We have 3 use cases we need to think about separately:

          1) A cluster with one permanent JT. This should have a single well know place it logs.

          2) A HOD JT, this should send logs to a user specified directory

          3) Central logging of job summary data that works in either of the above cases. We should create a distinct JIRA to discuss this.

          Show
          eric baldeschwieler added a comment - I don't think it makes sense to address HADOOP-1876 by storing history on HDFS by default. I think it is better to keep the server logs on the local machine by default and allow a configuration to store them in a well specified place on HDFS. This will keep a HOD user from creating leavings all over the HDFS, but allow for alternate uses like HADOOP-1876 . We have 3 use cases we need to think about separately: 1) A cluster with one permanent JT. This should have a single well know place it logs. 2) A HOD JT, this should send logs to a user specified directory 3) Central logging of job summary data that works in either of the above cases. We should create a distinct JIRA to discuss this.
          Hide
          Amareshwari Sriramadasu added a comment - - edited

          To address both the issues (HADOOP-1876 and this), we will store job history files in a centralized location like _

          {/logs/history/<jobtracker.hostname>_<user.name>_<jobid>}

          _ on DFS. This location will avoid name conficts among users and hostnames.
          The job history viewer(centralized) can display available history files sorted based on the creation time on the dfs and user can select the job he wants to view from there.

          For 1876, job status can be queried giving the jobid, hostname and username. The hostname can be inferred and the username if not specified will be ignored and the first occurence of the jobid string taken.

          Show
          Amareshwari Sriramadasu added a comment - - edited To address both the issues ( HADOOP-1876 and this), we will store job history files in a centralized location like _ {/logs/history/<jobtracker.hostname>_<user.name>_<jobid>} _ on DFS. This location will avoid name conficts among users and hostnames. The job history viewer(centralized) can display available history files sorted based on the creation time on the dfs and user can select the job he wants to view from there. For 1876, job status can be queried giving the jobid, hostname and username. The hostname can be inferred and the username if not specified will be ignored and the first occurence of the jobid string taken.
          Hide
          Runping Qi added a comment -

          BTW, Is this issue the same as H-1876
          (https://issues.apache.org/jira/browse/HADOOP-1876) ?

          Show
          Runping Qi added a comment - BTW, Is this issue the same as H-1876 ( https://issues.apache.org/jira/browse/HADOOP-1876 ) ?
          Hide
          Devaraj Das added a comment -

          +1 on Eric's suggestion. The jobhistory viewer (web server with the job history related JSPs) can take the output directory as the input and populate the history datastructures. This can be on a per-user basis for now (e.g., bin/hadoop jobhistoryview -output <dir> .. ), and, in the future, we could make the viewer a centralized web-enabled server that anyone can use. Thoughts?

          The output data may be deleted anytime when it is no longer needed. The log data may be needed long after the output data is deleted.

          As Eric suggested, this can be solved by allowing the user to move the files to some persistent location...

          Show
          Devaraj Das added a comment - +1 on Eric's suggestion. The jobhistory viewer (web server with the job history related JSPs) can take the output directory as the input and populate the history datastructures. This can be on a per-user basis for now (e.g., bin/hadoop jobhistoryview -output <dir> .. ), and, in the future, we could make the viewer a centralized web-enabled server that anyone can use. Thoughts? The output data may be deleted anytime when it is no longer needed. The log data may be needed long after the output data is deleted. As Eric suggested, this can be solved by allowing the user to move the files to some persistent location...
          Hide
          Runping Qi added a comment -

          The output data may be deleted anytime when it is no longer needed. The log data may be needed long after the output data is deleted.

          Show
          Runping Qi added a comment - The output data may be deleted anytime when it is no longer needed. The log data may be needed long after the output data is deleted.
          Hide
          eric baldeschwieler added a comment -

          Why copy files once a job is complete? Why not just always write them directly to the HDFS (or any other URL the user configures)?

          I'm not a real fan of hidden directories like these. The user will not know of them and potentially will fill a lot of disk/name space with never viewed material. I'd be much happier if job history were considered part of the output of a job, unless configured otherwise. IE put it in the map-reduce output directory in a file or directories prefixed with an underscore. So <output>/_jobHistory or perhaps <output>/_logs/history.

          We added the convention that map-reduce ignores underscore prefixed files specifically to allow this use case...

          This also reduces jobid/name confusion, since the history is directly associated with the job's output.

          We could then provide an option to put it in another location if the user desires.

          thoughts?

          Show
          eric baldeschwieler added a comment - Why copy files once a job is complete? Why not just always write them directly to the HDFS (or any other URL the user configures)? I'm not a real fan of hidden directories like these. The user will not know of them and potentially will fill a lot of disk/name space with never viewed material. I'd be much happier if job history were considered part of the output of a job, unless configured otherwise. IE put it in the map-reduce output directory in a file or directories prefixed with an underscore. So <output>/_jobHistory or perhaps <output>/_logs/history. We added the convention that map-reduce ignores underscore prefixed files specifically to allow this use case... This also reduces jobid/name confusion, since the history is directly associated with the job's output. We could then provide an option to put it in another location if the user desires. thoughts?

            People

            • Assignee:
              Amareshwari Sriramadasu
              Reporter:
              Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development