Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.7, 2.0.4-alpha
    • Fix Version/s: 2.1.0-beta, 0.23.9
    • Component/s: jobhistoryserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The history server can easily take many minutes to startup when there are a significant number of jobs to scan in the done directory. However the scanning of files is not the bottleneck, rather it's the heavy use of ConcurrentSkipListMap.size in HistoryFileManager.

      ConcurrentSkipListMap.size is a very expensive operation, especially on maps with many entries, as it has to scan every entry to compute the size. We should avoid calling this method or at least minimize its use.

      1. mr-5268.patch
        10 kB
        Karthik Kambatla
      2. mr-5268.patch
        11 kB
        Karthik Kambatla
      3. mr-5268.patch
        11 kB
        Karthik Kambatla
      4. mr-5268.patch
        10 kB
        Karthik Kambatla
      5. mr-5268.patch
        9 kB
        Karthik Kambatla
      6. mr-5268-prelim.patch
        5 kB
        Karthik Kambatla

        Activity

        Hide
        Jason Lowe added a comment -

        Karthik, do you have an ETA on this? We're particularly interested in getting a fix for this.

        Show
        Jason Lowe added a comment - Karthik, do you have an ETA on this? We're particularly interested in getting a fix for this.
        Hide
        Karthik Kambatla added a comment -

        Jason Lowe, I was planning to work on it today, but haven't started it yet. Let me know if you want to take it up. Otherwise, I ll try to get in atleast a preliminary patch late today or early tomorrow.

        Show
        Karthik Kambatla added a comment - Jason Lowe , I was planning to work on it today, but haven't started it yet. Let me know if you want to take it up. Otherwise, I ll try to get in atleast a preliminary patch late today or early tomorrow.
        Hide
        Jason Lowe added a comment -

        That will be fine, just making sure you were planning on getting something posted in the next few days. If something comes up and you won't get to it, I'd be happy to push it across the finish line. Thanks, Karthik!

        Show
        Jason Lowe added a comment - That will be fine, just making sure you were planning on getting something posted in the next few days. If something comes up and you won't get to it, I'd be happy to push it across the finish line. Thanks, Karthik!
        Hide
        Karthik Kambatla added a comment -

        The least intrusive way seems to be to maintain the size of the ConcurrentSkipListMap outside in JobListCache. Uploading a preliminary patch with the changes:

        1. Add an AtomicInteger field size to JobListcache, incremented when we add an entry and decremented when we remove
        2. Haven't used locks around cache#insert/delete and size update, as I think it might be okay to not conform to the maximum size exactly. If we choose to synchronize, we should probably use int for size instead of AtomicInteger.
        3. I am still working on tests for JobListCache, will try to update the patch tomorrow.

        IIUC, we are using ConcurrentSkipListMap now and TreeMap in one of the previous versions to evict items in the order of JobId. Is the ordering a requirement? If it is not, we could may be use something like GuavaCache? Thoughts?

        Show
        Karthik Kambatla added a comment - The least intrusive way seems to be to maintain the size of the ConcurrentSkipListMap outside in JobListCache . Uploading a preliminary patch with the changes: Add an AtomicInteger field size to JobListcache, incremented when we add an entry and decremented when we remove Haven't used locks around cache#insert/delete and size update, as I think it might be okay to not conform to the maximum size exactly. If we choose to synchronize, we should probably use int for size instead of AtomicInteger. I am still working on tests for JobListCache, will try to update the patch tomorrow. IIUC, we are using ConcurrentSkipListMap now and TreeMap in one of the previous versions to evict items in the order of JobId . Is the ordering a requirement? If it is not, we could may be use something like GuavaCache? Thoughts?
        Hide
        Karthik Kambatla added a comment -

        Updated patch with tests.

        Show
        Karthik Kambatla added a comment - Updated patch with tests.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12585168/mr-5268.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3686//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3686//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585168/mr-5268.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3686//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3686//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        Is the ordering a requirement? If it is not, we could may be use something like GuavaCache?

        Ordering is important for the code that evicts entries from the cache. The only other access that isn't a simple put/get is in CachedHistoryStorage.getAllPartialJobs, but that method explicitly builds a SortedMap separately and returns that.

        Therefore I think using a Guava cache is possible, although it wouldn't be as straightforward as it might initially seem. I think we'd have to use a weighted cache with the weight being an increasing sequence number so it removes entries based on cache size but in insertion order rather than LRU. Also we'd need a removal listener to replicate the special-case cache removal logic when a move is still pending or has failed, and have a way to follow up on history file infos that need to be flushed from the cache but the move is still pending. It's all possible, just not as simple as dropping in the Guava cache and declaring victory.

        I'm OK with the current approach with one suggestion: I'd prefer the size/collection pair implementation detail abstracted into a local utility class. That way as the code is maintained someone can't easily add or remove elements from the collection and forget to update the separate size variable.

        Show
        Jason Lowe added a comment - Is the ordering a requirement? If it is not, we could may be use something like GuavaCache? Ordering is important for the code that evicts entries from the cache. The only other access that isn't a simple put/get is in CachedHistoryStorage.getAllPartialJobs, but that method explicitly builds a SortedMap separately and returns that. Therefore I think using a Guava cache is possible, although it wouldn't be as straightforward as it might initially seem. I think we'd have to use a weighted cache with the weight being an increasing sequence number so it removes entries based on cache size but in insertion order rather than LRU. Also we'd need a removal listener to replicate the special-case cache removal logic when a move is still pending or has failed, and have a way to follow up on history file infos that need to be flushed from the cache but the move is still pending. It's all possible, just not as simple as dropping in the Guava cache and declaring victory. I'm OK with the current approach with one suggestion: I'd prefer the size/collection pair implementation detail abstracted into a local utility class. That way as the code is maintained someone can't easily add or remove elements from the collection and forget to update the separate size variable.
        Hide
        Karthik Kambatla added a comment -

        I think we'd have to use a weighted cache with the weight being an increasing sequence number so it removes entries based on cache size but in insertion order rather than LRU.

        Also, we will have to set the concurrency level to 1 to ensure the weighted cache follows the order.

        I'm OK with the current approach with one suggestion: I'd prefer the size/collection pair implementation detail abstracted into a local utility class.

        Makes total sense. Will do.

        Show
        Karthik Kambatla added a comment - I think we'd have to use a weighted cache with the weight being an increasing sequence number so it removes entries based on cache size but in insertion order rather than LRU. Also, we will have to set the concurrency level to 1 to ensure the weighted cache follows the order. I'm OK with the current approach with one suggestion: I'd prefer the size/collection pair implementation detail abstracted into a local utility class. Makes total sense. Will do.
        Hide
        Karthik Kambatla added a comment -

        Updated patch to abstract out collection/size into a local utility class and minimal changes to JobListCache.

        Considered implementing a full-blown implementation of ConcurrentSkipListMapWithSize extending ConcurrentSkipList and adding it to hadoop-common, but decided against it as that seemed a little overboard.

        Show
        Karthik Kambatla added a comment - Updated patch to abstract out collection/size into a local utility class and minimal changes to JobListCache. Considered implementing a full-blown implementation of ConcurrentSkipListMapWithSize extending ConcurrentSkipList and adding it to hadoop-common, but decided against it as that seemed a little overboard.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12585274/mr-5268.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3687//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3687//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585274/mr-5268.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3687//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3687//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        Considered implementing a full-blown implementation of ConcurrentSkipListMapWithSize extending ConcurrentSkipList and adding it to hadoop-common, but decided against it as that seemed a little overboard.

        Agreed. Actually I'm a bit surprised to see a templated interface for the private class. The whole purpose of the class is to be a map of JobId to HistoryFileInfo so templating seems a bit overkill. Not a big deal, though.

        I was a bit puzzled to see the synchronized methods instead of the AtomicInteger. What was the reasoning behind that change? It seems a bit odd to see a ConcurrentSkipListMap behind synchronized accesses.

        BTW, I built a local test of 50,000 jhist files with the history server configured to cache 20,000. Without this change the history server takes over 60 seconds to process the existing jobs, and after the patch it processes them in around 6 seconds.

        Show
        Jason Lowe added a comment - Considered implementing a full-blown implementation of ConcurrentSkipListMapWithSize extending ConcurrentSkipList and adding it to hadoop-common, but decided against it as that seemed a little overboard. Agreed. Actually I'm a bit surprised to see a templated interface for the private class. The whole purpose of the class is to be a map of JobId to HistoryFileInfo so templating seems a bit overkill. Not a big deal, though. I was a bit puzzled to see the synchronized methods instead of the AtomicInteger. What was the reasoning behind that change? It seems a bit odd to see a ConcurrentSkipListMap behind synchronized accesses. BTW, I built a local test of 50,000 jhist files with the history server configured to cache 20,000. Without this change the history server takes over 60 seconds to process the existing jobs, and after the patch it processes them in around 6 seconds.
        Hide
        Karthik Kambatla added a comment -

        Actually I'm a bit surprised to see a templated interface for the private class.

        I initially had a custom/ un-templated implementation, but added the template to limit the changes to JobListCache. Let me get rid of it.

        I was a bit puzzled to see the synchronized methods instead of the AtomicInteger. What was the reasoning behind that change?

        Agree using an AtomicInteger would be faster. However, my thinking was that given we are abstracting it out, someone working on JobListCache in the future might be oblivious to the potential race of mapSize not changing along with the collection. But, I suppose we could explicitly comment that behavior.

        Let me update the patch with the two changes.

        Show
        Karthik Kambatla added a comment - Actually I'm a bit surprised to see a templated interface for the private class. I initially had a custom/ un-templated implementation, but added the template to limit the changes to JobListCache. Let me get rid of it. I was a bit puzzled to see the synchronized methods instead of the AtomicInteger. What was the reasoning behind that change? Agree using an AtomicInteger would be faster. However, my thinking was that given we are abstracting it out, someone working on JobListCache in the future might be oblivious to the potential race of mapSize not changing along with the collection. But, I suppose we could explicitly comment that behavior. Let me update the patch with the two changes.
        Hide
        Karthik Kambatla added a comment -

        Updated the patch to address Jason's comments:

        1. use AtomicInteger instead of synchronized methods
        2. wrapper not to use templates
        Show
        Karthik Kambatla added a comment - Updated the patch to address Jason's comments: use AtomicInteger instead of synchronized methods wrapper not to use templates
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12585515/mr-5268.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3703//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3703//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585515/mr-5268.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3703//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3703//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        Thanks for updating so quickly, Karthik. Sorry, I should have been more clear on my nit with the templates. I was surprised to see such a generic type, both in name and template use, for a specific, private use-case. If the name is something so generic as ConcurrentSkipListMapWithSize then I think it is appropriate to use templates with that name since the name implies it is generic. If we think the templates are overkill then I think it's better to name it more specifically for its purpose, e.g.: JobHistoryInfoMap.

        I do like the AtomicInteger change. And as far as the size inconsistency concern, that's already an issue because of the very nature of concurrency whether we synchronize the methods or not. As soon as we get/compute the size, it could be wrong as someone else comes in and changes the map immediately afterwards. In this case the size is only being used to loosely cap the size of the job cache, and it's not critical if we're a few jobs over or a few jobs under that limit.

        So I think we're really close. I'd like to see either the ConcurrentSkipListMapWithSize name changed to be more specific to its use case or keep the name and have the templates restored. I don't care too much either way.

        Show
        Jason Lowe added a comment - Thanks for updating so quickly, Karthik. Sorry, I should have been more clear on my nit with the templates. I was surprised to see such a generic type, both in name and template use, for a specific, private use-case. If the name is something so generic as ConcurrentSkipListMapWithSize then I think it is appropriate to use templates with that name since the name implies it is generic. If we think the templates are overkill then I think it's better to name it more specifically for its purpose, e.g.: JobHistoryInfoMap. I do like the AtomicInteger change. And as far as the size inconsistency concern, that's already an issue because of the very nature of concurrency whether we synchronize the methods or not. As soon as we get/compute the size, it could be wrong as someone else comes in and changes the map immediately afterwards. In this case the size is only being used to loosely cap the size of the job cache, and it's not critical if we're a few jobs over or a few jobs under that limit. So I think we're really close. I'd like to see either the ConcurrentSkipListMapWithSize name changed to be more specific to its use case or keep the name and have the templates restored. I don't care too much either way.
        Hide
        Karthik Kambatla added a comment -

        Updated the helper class name to JobIdHistoryFileInfoMap

        Show
        Karthik Kambatla added a comment - Updated the helper class name to JobIdHistoryFileInfoMap
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12585626/mr-5268.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3709//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3709//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585626/mr-5268.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3709//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3709//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        Wow, that was fast! Unfortunately ConcurrentSkipListMapWithSize is still showing up in filenames.

        Also taking a closer look at the unit tests, FakeJobId should be removed and replaced with MRBuilderUtils.newJobId. There's a form of that method where we can simply pass numbers for the cluster timestamp, app ID, and job ID directly.

        Show
        Jason Lowe added a comment - Wow, that was fast! Unfortunately ConcurrentSkipListMapWithSize is still showing up in filenames. Also taking a closer look at the unit tests, FakeJobId should be removed and replaced with MRBuilderUtils.newJobId. There's a form of that method where we can simply pass numbers for the cluster timestamp, app ID, and job ID directly.
        Hide
        Karthik Kambatla added a comment -

        Thanks for the patient reviews, Jason.

        My bad forgot to rename the corresponding test - fixed that now. Also, the tests now use MRBuilderUtils instead of FakeJob.

        Reviewed the patch carefully, hopefully this takes care of everything.

        Show
        Karthik Kambatla added a comment - Thanks for the patient reviews, Jason. My bad forgot to rename the corresponding test - fixed that now. Also, the tests now use MRBuilderUtils instead of FakeJob. Reviewed the patch carefully, hopefully this takes care of everything.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12585646/mr-5268.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 2 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3711//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3711//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585646/mr-5268.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3711//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3711//console This message is automatically generated.
        Hide
        Jason Lowe added a comment -

        +1 for the latest patch.

        Thanks so much for the rapid turnarounds on the patches and your patience, Karthik!

        Show
        Jason Lowe added a comment - +1 for the latest patch. Thanks so much for the rapid turnarounds on the patches and your patience, Karthik!
        Hide
        Jason Lowe added a comment -

        Will commit this after the 2.0.5 / 2.1.0 dust has cleared which should be done this weekend.

        Show
        Jason Lowe added a comment - Will commit this after the 2.0.5 / 2.1.0 dust has cleared which should be done this weekend.
        Hide
        Jason Lowe added a comment -

        Thanks, Karthik! I committed this to trunk, branch-2, and branch-0.23.

        Show
        Jason Lowe added a comment - Thanks, Karthik! I committed this to trunk, branch-2, and branch-0.23.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-trunk-Commit #3837 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3837/)
        MAPREDUCE-5268. Improve history server startup performance. Contributed by Karthik Kambatla (Revision 1489012)

        Result = SUCCESS
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489012
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobIdHistoryFileInfoMap.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobListCache.java
        Show
        Hudson added a comment - Integrated in Hadoop-trunk-Commit #3837 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3837/ ) MAPREDUCE-5268 . Improve history server startup performance. Contributed by Karthik Kambatla (Revision 1489012) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489012 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobIdHistoryFileInfoMap.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobListCache.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Yarn-trunk #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/230/)
        MAPREDUCE-5268. Improve history server startup performance. Contributed by Karthik Kambatla (Revision 1489012)

        Result = FAILURE
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489012
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobIdHistoryFileInfoMap.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobListCache.java
        Show
        Hudson added a comment - Integrated in Hadoop-Yarn-trunk #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/230/ ) MAPREDUCE-5268 . Improve history server startup performance. Contributed by Karthik Kambatla (Revision 1489012) Result = FAILURE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489012 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobIdHistoryFileInfoMap.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobListCache.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #628 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/628/)
        svn merge -c 1489012 FIXES: MAPREDUCE-5268. Improve history server startup performance. Contributed by Karthik Kambatla (Revision 1489016)

        Result = UNSTABLE
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489016
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobIdHistoryFileInfoMap.java
        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobListCache.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #628 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/628/ ) svn merge -c 1489012 FIXES: MAPREDUCE-5268 . Improve history server startup performance. Contributed by Karthik Kambatla (Revision 1489016) Result = UNSTABLE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489016 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobIdHistoryFileInfoMap.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobListCache.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1420 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/)
        MAPREDUCE-5268. Improve history server startup performance. Contributed by Karthik Kambatla (Revision 1489012)

        Result = FAILURE
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489012
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobIdHistoryFileInfoMap.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobListCache.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1420 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1420/ ) MAPREDUCE-5268 . Improve history server startup performance. Contributed by Karthik Kambatla (Revision 1489012) Result = FAILURE jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489012 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobIdHistoryFileInfoMap.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobListCache.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1446 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/)
        MAPREDUCE-5268. Improve history server startup performance. Contributed by Karthik Kambatla (Revision 1489012)

        Result = SUCCESS
        jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489012
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobIdHistoryFileInfoMap.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobListCache.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1446 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1446/ ) MAPREDUCE-5268 . Improve history server startup performance. Contributed by Karthik Kambatla (Revision 1489012) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1489012 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryFileManager.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobIdHistoryFileInfoMap.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/TestJobListCache.java

          People

          • Assignee:
            Karthik Kambatla
            Reporter:
            Jason Lowe
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development