Hadoop Common
  1. Hadoop Common
  2. HADOOP-6502

DistributedFileSystem#listStatus is very slow when listing a directory with a size of 1300

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.20.0
    • Fix Version/s: 0.23.2
    • Component/s: util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Target Version/s:

      Description

      When listing a directory of around 1300 children, it takes hundreds of milliseconds. It turns out the slowdowness is caused by the change made by HADOOP-4187. The return value of listStatus is an array of FileStatus. When deserializing each element of the array, ReflectionUtils#newInstance(Class<T>, Configuration) is called and then calls setConf, which calls setJobConf. SetJobConf checks if JobConf is on the class path by calling Configuration#getClassByName. Even though Configuration#getClassByName tries to optimize the lookup using a cached map, but since JobConf is not in the class path, so it is not in the cache. Every checkup ends up calling Class.ForName which is very expensive. Deserializing an array of 1300 entries requires calling of Class#ForName 1300 times!

      1. 6502.patch
        1 kB
        Sharad Agarwal
      2. 6502_v2.patch
        1 kB
        Sharad Agarwal
      3. hadoop-6502-trunk.txt
        1 kB
        Todd Lipcon
      4. hadoop-6502-trunk.txt
        4 kB
        Todd Lipcon

        Issue Links

          Activity

          Hide
          Sharad Agarwal added a comment -

          We can do something like:

          +    if (conf.getBoolean("mapred.jobconf.notpresent", false)) {
          +      return;//classpath check already done and jobconf is not in classpath.
          +    }
               try {
                 Class<?> jobConfClass = 
                   conf.getClassByName("org.apache.hadoop.mapred.JobConf");
          @@ -86,9 +89,11 @@ public class ReflectionUtils {
                   Method configureMethod = 
                     jobConfigurableClass.getMethod("configure", jobConfClass);
                   configureMethod.invoke(theObject, conf);
          +        
                 }
               } catch (ClassNotFoundException e) {
                 //JobConf/JobConfigurable not in classpath. no need to configure
          +      conf.setBoolean("mapred.jobconf.notpresent", true);
               }
          

          I think the performance issue only comes if jobconf is not in classpath resulting in ClassNotFoundException on each check. Hairong, can you please verify if the above fix helps ?

          Show
          Sharad Agarwal added a comment - We can do something like: + if (conf.getBoolean( "mapred.jobconf.notpresent" , false )) { + return ; //classpath check already done and jobconf is not in classpath. + } try { Class <?> jobConfClass = conf.getClassByName( "org.apache.hadoop.mapred.JobConf" ); @@ -86,9 +89,11 @@ public class ReflectionUtils { Method configureMethod = jobConfigurableClass.getMethod( "configure" , jobConfClass); configureMethod.invoke(theObject, conf); + } } catch (ClassNotFoundException e) { //JobConf/JobConfigurable not in classpath. no need to configure + conf.setBoolean( "mapred.jobconf.notpresent" , true ); } I think the performance issue only comes if jobconf is not in classpath resulting in ClassNotFoundException on each check. Hairong, can you please verify if the above fix helps ?
          Hide
          Todd Lipcon added a comment -

          Is this dangerous? Will we ever have a case where the JobConf isn't in the classpath once, and then later shows up because the classloader changed?

          Show
          Todd Lipcon added a comment - Is this dangerous? Will we ever have a case where the JobConf isn't in the classpath once, and then later shows up because the classloader changed?
          Hide
          Doug Cutting added a comment -

          I don't think it's appropriate to cache JVM state in the configuration.

          I think we can better fix this by changing Configuration#getClassByName() to cache failures as well as successes.

          > Will we ever have a case where the JobConf isn't in the classpath once, and then later shows up because the classloader changed?

          The cache in Configuration is per-classloader. So as long as we go through that we should be safe.

          Show
          Doug Cutting added a comment - I don't think it's appropriate to cache JVM state in the configuration. I think we can better fix this by changing Configuration#getClassByName() to cache failures as well as successes. > Will we ever have a case where the JobConf isn't in the classpath once, and then later shows up because the classloader changed? The cache in Configuration is per-classloader. So as long as we go through that we should be safe.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          DistributedFileSystem#listStatus requires checking the JobConf class.

          The above statement sounds totally wrong to me. If mapreduce needs a special ReflectionUtils, why don't we add one in mapreduce?

          Show
          Tsz Wo Nicholas Sze added a comment - DistributedFileSystem#listStatus requires checking the JobConf class. The above statement sounds totally wrong to me. If mapreduce needs a special ReflectionUtils, why don't we add one in mapreduce?
          Hide
          Todd Lipcon added a comment -

          Nicholas: the problem is that ReflectionUtils.getInstance gets called by common code in Configuration. MapReduce wouldn't just need its own ReflectionUtils, it would also need its own Configuration, no?

          Show
          Todd Lipcon added a comment - Nicholas: the problem is that ReflectionUtils.getInstance gets called by common code in Configuration. MapReduce wouldn't just need its own ReflectionUtils, it would also need its own Configuration, no?
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Since JobConf extends Configuration, could JobConf override the common codes ?

          Show
          Tsz Wo Nicholas Sze added a comment - Since JobConf extends Configuration, could JobConf override the common codes ?
          Hide
          Todd Lipcon added a comment -

          Actually, after thinking some more, even that wouldn't work. There's lots of Common code that uses ReflectionUtils.newInstance, for example the serialziation stuff which instantiates writables. A user is free to make their writables JobConfigurable, etc. I don't think there's a particularly simple solution here.

          The cache in Configuration is per-classloader. So as long as we go through that we should be safe.

          If we make the assumption that classloaders never pick up new classes, that's true. But I don't think the JVM has a negative class cache, does it? That is, if you try to load a class when it doesn't exist, then move the class into the classpath and try to load again, it might pick it up.

          Show
          Todd Lipcon added a comment - Actually, after thinking some more, even that wouldn't work. There's lots of Common code that uses ReflectionUtils.newInstance, for example the serialziation stuff which instantiates writables. A user is free to make their writables JobConfigurable, etc. I don't think there's a particularly simple solution here. The cache in Configuration is per-classloader. So as long as we go through that we should be safe. If we make the assumption that classloaders never pick up new classes, that's true. But I don't think the JVM has a negative class cache, does it? That is, if you try to load a class when it doesn't exist, then move the class into the classpath and try to load again, it might pick it up.
          Hide
          Doug Cutting added a comment -

          > If we make the assumption that classloaders never pick up new classes, that's true.

          I am happy to make that assumption. I don't think dynamically adding classes to a class path as an application runs is a pattern we need to support. Consider the case where you place a different definition earlier in the classpath. No cache would pick that up. Caching negatives seems like a very small step beyond that.

          Show
          Doug Cutting added a comment - > If we make the assumption that classloaders never pick up new classes, that's true. I am happy to make that assumption. I don't think dynamically adding classes to a class path as an application runs is a pattern we need to support. Consider the case where you place a different definition earlier in the classpath. No cache would pick that up. Caching negatives seems like a very small step beyond that.
          Hide
          Todd Lipcon added a comment -

          I don't think dynamically adding classes to a class path as an application runs is a pattern we need to support.

          I tend to agree. Steve Loughran - you out there? I think you probably are the one doing the wackiest stuff with classloaders and Hadoop

          Show
          Todd Lipcon added a comment - I don't think dynamically adding classes to a class path as an application runs is a pattern we need to support. I tend to agree. Steve Loughran - you out there? I think you probably are the one doing the wackiest stuff with classloaders and Hadoop
          Hide
          steve_l added a comment -

          Disclaimer I have long advocated having a ASF exam in classloaders; nobody who hasn't passed the exam would be allowed to mess with classloaders in any apache project. As there is no such exam, there is no proof that I can be considered competent enough to do this, and you should treat everything I say with caution. Test my statements, preferably in JUnit methods.

          1. Adding new classes is generally rare unless you are running something that is generating java classes on the fly; JSP compilers do this. Even then, they try not to mess around with things higher up the hierarchy (exception, JBoss default classloader, the one that's broken that everyone hates).

          2. Modern, OSGi-style classloaders are fairly strict, I don't think they add stuff higher up. More of a general concern when you play with classloader trees are

          1. it's easy to leak classloaders. retain one ref to a class loaded by a child classloader and the classloader never gets GC'd, doesn't pick up
            updated JARs, consumes memory, stops your build overwriting any locked
            JARs (windows only)
          2. all the rules about singletons and equality goes out the window.

          3. I would go for caching the failure. For those people playing games with classloaders, tough. But do note that if the JSP engine does need to
          compile a JSP class, then Hadoop is adding classes to some classpath
          in the Hadoop JVM. So your tools may be doing what you don't think is
          happening, even on a "normal" Hadoop instance.

          4. Looking at the code in more detail, the things a bit of an ugly hack, a contrived workaround to avoid a cycle. If there was an elegant solution to this that didn't evolve reflection, things would be much better. Nothing obvious springs to mind.

          Show
          steve_l added a comment - Disclaimer I have long advocated having a ASF exam in classloaders; nobody who hasn't passed the exam would be allowed to mess with classloaders in any apache project. As there is no such exam, there is no proof that I can be considered competent enough to do this, and you should treat everything I say with caution. Test my statements, preferably in JUnit methods. 1. Adding new classes is generally rare unless you are running something that is generating java classes on the fly; JSP compilers do this. Even then, they try not to mess around with things higher up the hierarchy (exception, JBoss default classloader, the one that's broken that everyone hates). 2. Modern, OSGi-style classloaders are fairly strict, I don't think they add stuff higher up. More of a general concern when you play with classloader trees are it's easy to leak classloaders. retain one ref to a class loaded by a child classloader and the classloader never gets GC'd, doesn't pick up updated JARs, consumes memory, stops your build overwriting any locked JARs (windows only) all the rules about singletons and equality goes out the window. 3. I would go for caching the failure. For those people playing games with classloaders, tough. But do note that if the JSP engine does need to compile a JSP class, then Hadoop is adding classes to some classpath in the Hadoop JVM. So your tools may be doing what you don't think is happening, even on a "normal" Hadoop instance. 4. Looking at the code in more detail, the things a bit of an ugly hack, a contrived workaround to avoid a cycle. If there was an elegant solution to this that didn't evolve reflection, things would be much better. Nothing obvious springs to mind.
          Hide
          Tom White added a comment -

          > Looking at the code in more detail, the things a bit of an ugly hack, a contrived workaround to avoid a cycle.

          Remember that this workaround only needs to be supported until the deprecated JobConf class is removed, since the new API uses Configuration in its place.

          Show
          Tom White added a comment - > Looking at the code in more detail, the things a bit of an ugly hack, a contrived workaround to avoid a cycle. Remember that this workaround only needs to be supported until the deprecated JobConf class is removed, since the new API uses Configuration in its place.
          Hide
          Arun C Murthy added a comment -

          I had a discussion with Hairong about this one...

          The important thing to note is that listStatus as described in this jira iff we are writing a stand-alone DFSClient without map-reduce jars in the classpath of the application. It is so because if JobConf/JobConfigurable are in the classpath the look-up is done only once and is cached in Configuration.getClassByName. I believe Hairong was running a stand-alone test where she did not have MR jars on the classpath.

          Unfortunately, we cannot remove this feature from ReflectionUtils without breaking applications. I'd propose we fix it by removing the offending code at the same time we remove the deprecated org.apache.hadoop.mapred package. Until then it affects a small minority of applications... even bin/hadoop commands are 'ok' since MR jars are on the classpath.

          However bin/hdfs might have this problem... sigh.

          Show
          Arun C Murthy added a comment - I had a discussion with Hairong about this one... The important thing to note is that listStatus as described in this jira iff we are writing a stand-alone DFSClient without map-reduce jars in the classpath of the application. It is so because if JobConf/JobConfigurable are in the classpath the look-up is done only once and is cached in Configuration.getClassByName. I believe Hairong was running a stand-alone test where she did not have MR jars on the classpath. Unfortunately, we cannot remove this feature from ReflectionUtils without breaking applications. I'd propose we fix it by removing the offending code at the same time we remove the deprecated org.apache.hadoop.mapred package. Until then it affects a small minority of applications... even bin/hadoop commands are 'ok' since MR jars are on the classpath. However bin/hdfs might have this problem... sigh.
          Hide
          Doug Cutting added a comment -

          Arun, do you see a problem with caching negatives in Configuration.getClassByName()? This seems like it should fix this and perhaps other related issues.

          Show
          Doug Cutting added a comment - Arun, do you see a problem with caching negatives in Configuration.getClassByName()? This seems like it should fix this and perhaps other related issues.
          Hide
          Arun C Murthy added a comment -

          I don't see much of a problem with caching negatives, but I'm not sure we need to anything here at all... I can go either way. shrug

          Show
          Arun C Murthy added a comment - I don't see much of a problem with caching negatives, but I'm not sure we need to anything here at all... I can go either way. shrug
          Hide
          steve_l added a comment -

          you could always just cache the special case of JobConf not found, a single flag rather than adding special not-found entries to the weak<name, class> hashmap. Then it becomes something that can be pulled when jobconf goes away, rather than another feature that needs to be retained forever because of the risk of other code relying on it.

          Show
          steve_l added a comment - you could always just cache the special case of JobConf not found, a single flag rather than adding special not-found entries to the weak<name, class> hashmap. Then it becomes something that can be pulled when jobconf goes away, rather than another feature that needs to be retained forever because of the risk of other code relying on it.
          Hide
          Hairong Kuang added a comment -

          > but I'm not sure we need to anything here at all...
          From the dfs point of view, of course, this should be fixed. It seems so weird that map/reduce related stuff would effect the performance of hdfs.

          +1 caching negatives. This should in general improve newInstance performance in the failure case.

          Show
          Hairong Kuang added a comment - > but I'm not sure we need to anything here at all... From the dfs point of view, of course, this should be fixed. It seems so weird that map/reduce related stuff would effect the performance of hdfs. +1 caching negatives. This should in general improve newInstance performance in the failure case.
          Hide
          Sharad Agarwal added a comment -

          Seems most folks are ok with caching negatives. I will post the patch shortly.

          Show
          Sharad Agarwal added a comment - Seems most folks are ok with caching negatives. I will post the patch shortly.
          Hide
          Sanjay Radia added a comment -

          +1 on caching negatives.

          Show
          Sanjay Radia added a comment - +1 on caching negatives.
          Hide
          Sharad Agarwal added a comment -

          Can anyone please review the attached patch ? Thanks

          Show
          Sharad Agarwal added a comment - Can anyone please review the attached patch ? Thanks
          Hide
          Tom White added a comment -

          Looks good to me (but I would suggest that you test it on the original problematic directory to see that it solves the problem). Also, please supply the class name to the ClassNotFoundException so users get better diagnostics.

          Show
          Tom White added a comment - Looks good to me (but I would suggest that you test it on the original problematic directory to see that it solves the problem). Also, please supply the class name to the ClassNotFoundException so users get better diagnostics.
          Hide
          Sharad Agarwal added a comment -

          Thanks Tom for looking at it. Here is the attached patch passing the class name in ClassNotFoundException.

          Show
          Sharad Agarwal added a comment - Thanks Tom for looking at it. Here is the attached patch passing the class name in ClassNotFoundException.
          Hide
          Suresh Srinivas added a comment -

          This is no longer an issue for trunk, since we have moved over to protocol buffers from Writables.

          Show
          Suresh Srinivas added a comment - This is no longer an issue for trunk, since we have moved over to protocol buffers from Writables.
          Hide
          Todd Lipcon added a comment -

          I believe this is still an issue in trunk, since the protobufs are still tunneled over a Writable-based mechanism. I see the following trace in an IPC benchmark I'm working on:

          "IPC Client (1065524847) connection to /127.0.0.1:12345 from todd" daemon prio=10 tid=0x000000000250e000 nid=0x3dba runnable [0x00007f96164f0000]
             java.lang.Thread.State: RUNNABLE
                  at java.util.zip.ZipFile.getEntry(Native Method)
                  at java.util.zip.ZipFile.getEntry(ZipFile.java:166)
                  - locked <0x00000007840bb5b0> (a java.util.jar.JarFile)
                  at java.util.jar.JarFile.getEntry(JarFile.java:223)
                  at java.util.jar.JarFile.getJarEntry(JarFile.java:206)
                  at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:771)
                  at sun.misc.URLClassPath.getResource(URLClassPath.java:185)
                  at java.net.URLClassLoader$1.run(URLClassLoader.java:209)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
                  - locked <0x0000000784000150> (a sun.misc.Launcher$AppClassLoader)
                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
                  - locked <0x0000000784000150> (a sun.misc.Launcher$AppClassLoader)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
                  at java.lang.Class.forName0(Native Method)
                  at java.lang.Class.forName(Class.java:264)
                  at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1162)
                  at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:89)
                  at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
                  at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
                  at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:835)
                  at org.apache.hadoop.ipc.Client$Connection.run(Client.java:762)
          
          Show
          Todd Lipcon added a comment - I believe this is still an issue in trunk, since the protobufs are still tunneled over a Writable-based mechanism. I see the following trace in an IPC benchmark I'm working on: "IPC Client (1065524847) connection to /127.0.0.1:12345 from todd" daemon prio=10 tid=0x000000000250e000 nid=0x3dba runnable [0x00007f96164f0000] java.lang. Thread .State: RUNNABLE at java.util.zip.ZipFile.getEntry(Native Method) at java.util.zip.ZipFile.getEntry(ZipFile.java:166) - locked <0x00000007840bb5b0> (a java.util.jar.JarFile) at java.util.jar.JarFile.getEntry(JarFile.java:223) at java.util.jar.JarFile.getJarEntry(JarFile.java:206) at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:771) at sun.misc.URLClassPath.getResource(URLClassPath.java:185) at java.net.URLClassLoader$1.run(URLClassLoader.java:209) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:205) at java.lang. ClassLoader .loadClass( ClassLoader .java:321) - locked <0x0000000784000150> (a sun.misc.Launcher$AppClassLoader) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) - locked <0x0000000784000150> (a sun.misc.Launcher$AppClassLoader) at java.lang. ClassLoader .loadClass( ClassLoader .java:266) at java.lang. Class .forName0(Native Method) at java.lang. Class .forName( Class .java:264) at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1162) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:89) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:835) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:762)
          Hide
          Todd Lipcon added a comment -

          attached patch is the same, but rebased against trunk.

          In my IPC benchmark, this raised the result from 2000 RPC/sec to 30000 RPC/sec.

          Show
          Todd Lipcon added a comment - attached patch is the same, but rebased against trunk. In my IPC benchmark, this raised the result from 2000 RPC/sec to 30000 RPC/sec.
          Hide
          Todd Lipcon added a comment -

          Assigning to myself to get this committed - will reassign to Hairong upon commit since she's the contributor, but this way it'll stay on my queue

          Show
          Todd Lipcon added a comment - Assigning to myself to get this committed - will reassign to Hairong upon commit since she's the contributor, but this way it'll stay on my queue
          Hide
          Todd Lipcon added a comment -

          New version of the patch has a slight improvement, such that ReflectionUtils.newInstance can avoid the construction of an exception, which is fairly expensive. This yielded a 7% improvement on my IPC benchmark beyond what the original patch did.

          Show
          Todd Lipcon added a comment - New version of the patch has a slight improvement, such that ReflectionUtils.newInstance can avoid the construction of an exception, which is fairly expensive. This yielded a 7% improvement on my IPC benchmark beyond what the original patch did.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12514444/hadoop-6502-trunk.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/595//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/595//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12514444/hadoop-6502-trunk.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/595//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/595//console This message is automatically generated.
          Hide
          Todd Lipcon added a comment -

          -1 tests included. The patch doesn't appear to include any new or modified tests.

          This is a performance fix. As indicated above, the IPC benchmark from HADOOP-8070 went from ~2k RPC/sec to ~30k RPC/sec with this patch.

          Also, I checked the classpath of a running NN and noted that it does not have the MR jars on its classpath. So, this bug does affect actual NNs in 0.23+

          Show
          Todd Lipcon added a comment - -1 tests included. The patch doesn't appear to include any new or modified tests. This is a performance fix. As indicated above, the IPC benchmark from HADOOP-8070 went from ~2k RPC/sec to ~30k RPC/sec with this patch. Also, I checked the classpath of a running NN and noted that it does not have the MR jars on its classpath. So, this bug does affect actual NNs in 0.23+
          Hide
          Aaron T. Myers added a comment -

          Tiny nit: In "} else {//check already performed on this class name" please put a space between "{" and "//".

          Otherwise the patch looks good to me. +1.

          Show
          Aaron T. Myers added a comment - Tiny nit: In "} else {//check already performed on this class name" please put a space between "{" and "//". Otherwise the patch looks good to me. +1.
          Hide
          Todd Lipcon added a comment -

          Committed to branch 0.23 and trunk. Thanks Sharad and all those who reviewed. I fixed the style nit on commit.

          Show
          Todd Lipcon added a comment - Committed to branch 0.23 and trunk. Thanks Sharad and all those who reviewed. I fixed the style nit on commit.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #1803 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1803/)
          HADOOP-6502. Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244620)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244620
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1803 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1803/ ) HADOOP-6502 . Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244620) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244620 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-0.23-Commit #553 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/553/)
          HADOOP-6502. Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244619)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244619
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-0.23-Commit #553 (See https://builds.apache.org/job/Hadoop-Common-0.23-Commit/553/ ) HADOOP-6502 . Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244619) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244619 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Commit #540 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/540/)
          HADOOP-6502. Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244619)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244619
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Commit #540 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/540/ ) HADOOP-6502 . Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244619) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244619 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #1729 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1729/)
          HADOOP-6502. Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244620)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244620
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1729 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1729/ ) HADOOP-6502 . Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244620) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244620 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #1741 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1741/)
          HADOOP-6502. Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244620)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244620
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1741 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1741/ ) HADOOP-6502 . Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244620) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244620 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Commit #557 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/557/)
          HADOOP-6502. Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244619)

          Result = ABORTED
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244619
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Commit #557 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/557/ ) HADOOP-6502 . Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244619) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244619 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #170 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/170/)
          HADOOP-6502. Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244619)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244619
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #170 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/170/ ) HADOOP-6502 . Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244619) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244619 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #957 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/957/)
          HADOOP-6502. Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244620)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244620
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #957 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/957/ ) HADOOP-6502 . Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244620) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244620 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-0.23-Build #198 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/198/)
          HADOOP-6502. Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244619)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244619
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #198 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/198/ ) HADOOP-6502 . Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244619) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244619 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #992 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/992/)
          HADOOP-6502. Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244620)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244620
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #992 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/992/ ) HADOOP-6502 . Improve the performance of Configuration.getClassByName when the class is not found by caching negative results. Contributed by Sharad Agarwal and Todd Lipcon. (Revision 1244620) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1244620 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java

            People

            • Assignee:
              Sharad Agarwal
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development