Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4705

Historyserver links expire before the history data does

    Details

      Description

      The historyserver can serve up links to jobs that become useless well before the job history files are purged. For example on a large, heavily used cluster we can end up rotating through the maximum number of jobs the historyserver can track fairly quickly. If a user was investigating an issue with a job using a saved historyserver URL, that URL can become useless because the historyserver has forgotten about the job even though the history files are still sitting in HDFS.

      We can tell the historyserver to keep track of more jobs by increasing mapreduce.jobhistory.joblist.cache.size, but this has a direct impact on the responsiveness of the main historyserver page since it serves up all the entries to the client at once. It looks like Hadoop 1.x avoided this issue by encoding the history file location into the URLs served up by the historyserver, so it didn't have to track a mapping between job ID and history file location.

        Activity

        Hide
        Siddharth Seth added a comment -

        The history server is supposed to be able to serve out older jobs by scanning through job directories (HistoryFileManager.scanOldDirsForJob) if a job is not found in the jobListcache. The number of directories it is aware of is controlled by "mapreduce.jobhistory.datestring.cache.size" - which defaults to a reasonably high value.
        Maybe this is broken at the moment ?, or we're trying to access specific jobs using a list API of some kind.

        Show
        Siddharth Seth added a comment - The history server is supposed to be able to serve out older jobs by scanning through job directories ( HistoryFileManager.scanOldDirsForJob ) if a job is not found in the jobListcache. The number of directories it is aware of is controlled by "mapreduce.jobhistory.datestring.cache.size" - which defaults to a reasonably high value. Maybe this is broken at the moment ?, or we're trying to access specific jobs using a list API of some kind.
        Hide
        Ravi Prakash added a comment -

        Jason got to it, before I could

        Show
        Ravi Prakash added a comment - Jason got to it, before I could
        Hide
        Jason Lowe added a comment -

        The problem occurs because HistoryFileManager.scanOldDirsForJob is using a job's serial number directly to lookup the serial number index (which would require an entry in the map per job) rather than the result of JobHistoryUtils.serialNumberDirectoryComponent which carves off the lower digits of the serial number, effectively grouping them into buckets of 1000. I'll post a patch shortly.

        Show
        Jason Lowe added a comment - The problem occurs because HistoryFileManager.scanOldDirsForJob is using a job's serial number directly to lookup the serial number index (which would require an entry in the map per job) rather than the result of JobHistoryUtils.serialNumberDirectoryComponent which carves off the lower digits of the serial number, effectively grouping them into buckets of 1000. I'll post a patch shortly.
        Hide
        Jason Lowe added a comment -

        Patch to fix scanOldDirsForJob to use a proper key for the serial number index lookup.

        Show
        Jason Lowe added a comment - Patch to fix scanOldDirsForJob to use a proper key for the serial number index lookup.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12548297/MAPREDUCE-4705.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2916//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2916//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12548297/MAPREDUCE-4705.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2916//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2916//console This message is automatically generated.
        Hide
        Siddharth Seth added a comment -

        +1. Committing this.

        Show
        Siddharth Seth added a comment - +1. Committing this.
        Hide
        Siddharth Seth added a comment -

        Committed to trunk, branch-2 and branch-0.23. Thanks Jason

        Show
        Siddharth Seth added a comment - Committed to trunk, branch-2 and branch-0.23. Thanks Jason
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2895 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2895/)
        MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2895 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2895/ ) MAPREDUCE-4705 . Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2833 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2833/)
        MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2833 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2833/ ) MAPREDUCE-4705 . Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2856 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2856/)
        MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850)

        Result = FAILURE
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2856 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2856/ ) MAPREDUCE-4705 . Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850) Result = FAILURE sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #399 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/399/)
        merge MAPREDUCE-4705 from trunk. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395852)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395852
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #399 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/399/ ) merge MAPREDUCE-4705 from trunk. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395852) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395852 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1190/)
        MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1190 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1190/ ) MAPREDUCE-4705 . Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1221 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1221/)
        MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850)

        Result = SUCCESS
        sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1221 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1221/ ) MAPREDUCE-4705 . Fix a bug in job history lookup, which makes older jobs inaccessible despite the presence of a valid history file. (Contributed by Jason Lowe) (Revision 1395850) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1395850 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobHistoryParsing.java

          People

          • Assignee:
            Jason Lowe
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development