Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8865

Improve quota initialization performance

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      After replaying edits, the whole file system tree is recursively scanned in order to initialize the quota. For big name space, this can take a very long time. Since this is done during namenode failover, it also affects failover latency.

      By using the Fork-Join framework, I was able to greatly reduce the initialization time. The following is the test result using the fsimage from one of the big name nodes we have.

      threads seconds
      1 (existing) 55
      1 (fork-join) 68
      4 16
      8 8
      12 6
      16 5
      20 4
      1. HDFS-8865_branch-2.6.patch
        11 kB
        Nicolas Fraison
      2. HDFS-8865_branch-2.7.patch
        18 kB
        Kihwal Lee
      3. HDFS-8865.patch
        18 kB
        Kihwal Lee
      4. HDFS-8865.v2.checkstyle.patch
        18 kB
        Kihwal Lee
      5. HDFS-8865.v2.patch
        18 kB
        Kihwal Lee
      6. HDFS-8865.v3.patch
        18 kB
        Kihwal Lee

        Issue Links

          Activity

          Hide
          kihwal Kihwal Lee added a comment -

          I will post the patch shortly.

          Show
          kihwal Kihwal Lee added a comment - I will post the patch shortly.
          Hide
          kihwal Kihwal Lee added a comment -

          While porting the patch from 2.6 to trunk, I noticed something odd in FSImagte.java.

                final EnumCounters<StorageType> typeSpaces =
                    new EnumCounters<StorageType>(StorageType.class);
                for (StorageType t : StorageType.getTypesSupportingQuota()) {
                  final long typeSpace = counts.getTypeSpaces().get(t) -
                      parentTypeSpaces.get(t);
                  final long typeQuota = q.getTypeSpaces().get(t);
                  if (Quota.isViolated(typeQuota, typeSpace)) {
                    LOG.warn("Storage type quota violation in image for "
                        + dir.getFullPathName()
                        + " type = " + t.toString() + " quota = "
                        + typeQuota + " < consumed " + typeSpace);
                  }
                }
          
                dir.getDirectoryWithQuotaFeature().setSpaceConsumed(namespace, ssConsumed,
                    typeSpaces);
          

          typeSpaces is passed to setSpaceConsumed, but it is empty. On restart of NN, storage type-based quota usage tracking will be reset.

          Show
          kihwal Kihwal Lee added a comment - While porting the patch from 2.6 to trunk, I noticed something odd in FSImagte.java . final EnumCounters<StorageType> typeSpaces = new EnumCounters<StorageType>(StorageType.class); for (StorageType t : StorageType.getTypesSupportingQuota()) { final long typeSpace = counts.getTypeSpaces().get(t) - parentTypeSpaces.get(t); final long typeQuota = q.getTypeSpaces().get(t); if (Quota.isViolated(typeQuota, typeSpace)) { LOG.warn( "Storage type quota violation in image for " + dir.getFullPathName() + " type = " + t.toString() + " quota = " + typeQuota + " < consumed " + typeSpace); } } dir.getDirectoryWithQuotaFeature().setSpaceConsumed(namespace, ssConsumed, typeSpaces); typeSpaces is passed to setSpaceConsumed , but it is empty. On restart of NN, storage type-based quota usage tracking will be reset.
          Hide
          xyao Xiaoyu Yao added a comment -

          Kihwal Lee, thanks for working on this improvement work and fixing the issue on quota by storage type usage update.
          Can we fix the quota by storage type update issue in a separate JIRA?

          Show
          xyao Xiaoyu Yao added a comment - Kihwal Lee , thanks for working on this improvement work and fixing the issue on quota by storage type usage update. Can we fix the quota by storage type update issue in a separate JIRA?
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 26m 11s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 8m 59s There were no new javac warning messages.
          +1 javadoc 10m 20s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 55s The applied patch generated 10 new checkstyle issues (total was 491, now 498).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          -1 findbugs 2m 44s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings.
          +1 native 3m 9s Pre-build of native portion
          -1 hdfs tests 175m 32s Tests failed in hadoop-hdfs.
              231m 23s  



          Reason Tests
          FindBugs module:hadoop-hdfs
          Failed unit tests hadoop.hdfs.server.namenode.snapshot.TestSnapshot
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
          Timed out tests org.apache.hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749347/HDFS-8865.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8f73bdd
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11938/testReport/
          Java 1.7.0_55
          uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11938/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 26m 11s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 8m 59s There were no new javac warning messages. +1 javadoc 10m 20s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 55s The applied patch generated 10 new checkstyle issues (total was 491, now 498). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. -1 findbugs 2m 44s The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. +1 native 3m 9s Pre-build of native portion -1 hdfs tests 175m 32s Tests failed in hadoop-hdfs.     231m 23s   Reason Tests FindBugs module:hadoop-hdfs Failed unit tests hadoop.hdfs.server.namenode.snapshot.TestSnapshot   hadoop.hdfs.TestLeaseRecovery2   hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749347/HDFS-8865.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8f73bdd checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11938/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11938/testReport/ Java 1.7.0_55 uname Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11938/console This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment - - edited

          New patch

          • Fixed the findbug warning of unused variable.
          • Fixed the test breakages. Snapshot tests were breaking because of casting ref inode to dir. Changed it to use child.asDirectory() when generating work. TestLeaseRecovery2 and TestHDFSCLI are also passing locally.
          • Addressed most checkstyle issues. A couple of them are ignored.
          Show
          kihwal Kihwal Lee added a comment - - edited New patch Fixed the findbug warning of unused variable. Fixed the test breakages. Snapshot tests were breaking because of casting ref inode to dir. Changed it to use child.asDirectory() when generating work. TestLeaseRecovery2 and TestHDFSCLI are also passing locally. Addressed most checkstyle issues. A couple of them are ignored.
          Hide
          kihwal Kihwal Lee added a comment - - edited

          Can we fix the quota by storage type update issue in a separate JIRA?

          It is kind of hard to not do it, since that part of code is being modified. I will just do whatever seems natural in the new code. We can make sure it is fixed properly and improve the test case in HDFS-8879. We can set this jira as a dependency of HDFS-8879, if it ends up containing the fix.

          Show
          kihwal Kihwal Lee added a comment - - edited Can we fix the quota by storage type update issue in a separate JIRA? It is kind of hard to not do it, since that part of code is being modified. I will just do whatever seems natural in the new code. We can make sure it is fixed properly and improve the test case in HDFS-8879 . We can set this jira as a dependency of HDFS-8879 , if it ends up containing the fix.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Kihwal Lee! I agree with your comments above and post a patch on HDFS-8879. This will help us to backport these fixes to 2.7 branches.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Kihwal Lee ! I agree with your comments above and post a patch on HDFS-8879 . This will help us to backport these fixes to 2.7 branches.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 40s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 42s There were no new javac warning messages.
          +1 javadoc 9m 39s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 21s The applied patch generated 3 new checkstyle issues (total was 491, now 491).
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 20s mvn install still works.
          +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse.
          +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 4s Pre-build of native portion
          -1 hdfs tests 176m 53s Tests failed in hadoop-hdfs.
              221m 11s  



          Reason Tests
          Timed out tests org.apache.hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749585/HDFS-8865.v2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 8f73bdd
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11951/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11951/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11951/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11951/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 40s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 9m 39s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 21s The applied patch generated 3 new checkstyle issues (total was 491, now 491). +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 35s The patch built with eclipse:eclipse. +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 4s Pre-build of native portion -1 hdfs tests 176m 53s Tests failed in hadoop-hdfs.     221m 11s   Reason Tests Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749585/HDFS-8865.v2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 8f73bdd checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11951/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11951/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11951/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11951/console This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          Missed the one checkstyle warning.

          Show
          kihwal Kihwal Lee added a comment - Missed the one checkstyle warning.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 9s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 41s There were no new javac warning messages.
          +1 javadoc 9m 56s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 21s The applied patch generated 1 new checkstyle issues (total was 493, now 491).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 21s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 10s Pre-build of native portion
          -1 hdfs tests 186m 4s Tests failed in hadoop-hdfs.
              230m 19s  



          Reason Tests
          Timed out tests org.apache.hadoop.cli.TestHDFSCLI



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12749856/HDFS-8865.v2.checkstyle.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / fa1d84a
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11962/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11962/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11962/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11962/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 9s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 41s There were no new javac warning messages. +1 javadoc 9m 56s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 21s The applied patch generated 1 new checkstyle issues (total was 493, now 491). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 21s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 38s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 10s Pre-build of native portion -1 hdfs tests 186m 4s Tests failed in hadoop-hdfs.     230m 19s   Reason Tests Timed out tests org.apache.hadoop.cli.TestHDFSCLI Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12749856/HDFS-8865.v2.checkstyle.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / fa1d84a checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/11962/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11962/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11962/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11962/console This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          The checkstyle error is for the new config key, which I am not going to fix.
          The unit test timeout does not happen when I run it. Looks like it is failing in other pre-commit builds too, so it is not being caused by this patch.

          Show
          kihwal Kihwal Lee added a comment - The checkstyle error is for the new config key, which I am not going to fix. The unit test timeout does not happen when I run it. Looks like it is failing in other pre-commit builds too, so it is not being caused by this patch.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks for the patch, Kihwal Lee! It looks pretty good to me.

          Just a few comments:
          1. The number for large namespace looks impressive. Do you have the number for small/medium namespace?

          2. Is it possible to add some profiling info between these logs below so that we can easily find how long it takes to finish quota initialization from the log?

          LOG.info("Initializing quota with " + threads + " thread(s)");
          
          ...
          LOG.info("Quota initialization complete.\n" + counts);
          

          3. Can you change to parameterized logging to avoid parameter construction in case the log statement is disabled. For example,

          LOG.debug("Setting quota for {} +\n{}", dir,  myCounts);
          

          4. NIT: typo chached -> cached?

          // Directly access the name system to obtain the current chached usage.
          

          5. Now that HDFS-8879 is in, can you rebase and update the patch? Thanks!

          Show
          xyao Xiaoyu Yao added a comment - Thanks for the patch, Kihwal Lee ! It looks pretty good to me. Just a few comments: 1. The number for large namespace looks impressive. Do you have the number for small/medium namespace? 2. Is it possible to add some profiling info between these logs below so that we can easily find how long it takes to finish quota initialization from the log? LOG.info( "Initializing quota with " + threads + " thread(s)" ); ... LOG.info( "Quota initialization complete.\n" + counts); 3. Can you change to parameterized logging to avoid parameter construction in case the log statement is disabled. For example, LOG.debug( "Setting quota for {} +\n{}" , dir, myCounts); 4. NIT: typo chached -> cached? // Directly access the name system to obtain the current chached usage. 5. Now that HDFS-8879 is in, can you rebase and update the patch? Thanks!
          Hide
          kihwal Kihwal Lee added a comment -

          Thanks for the review Xiaoyu Yao.
          1. Numbers for small/medium size name space: I do not have numbers for them, but I expect them to be better than before. The test cases show it taking a couple of milliseconds for a very small name space, so the overhead of fork-join doesn't seem to be of too much concern. For very small namespaces, the ones that take far less than a second to initialize, the multithreaded initialization cpuld take a bit longer.

          2. This involves conversion to slf4j. I started down this path, but ended up needing to update IOUtils as well. Since it is not a matter of updating just FSImage, I won't do it in this jira. Instead I will simply wrap it with isDebugEnabled() check for now.

          Show
          kihwal Kihwal Lee added a comment - Thanks for the review Xiaoyu Yao . 1. Numbers for small/medium size name space: I do not have numbers for them, but I expect them to be better than before. The test cases show it taking a couple of milliseconds for a very small name space, so the overhead of fork-join doesn't seem to be of too much concern. For very small namespaces, the ones that take far less than a second to initialize, the multithreaded initialization cpuld take a bit longer. 2. This involves conversion to slf4j. I started down this path, but ended up needing to update IOUtils as well. Since it is not a matter of updating just FSImage , I won't do it in this jira. Instead I will simply wrap it with isDebugEnabled() check for now.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 50s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 9m 22s There were no new javac warning messages.
          +1 javadoc 11m 41s There were no new javadoc warning messages.
          +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 42s The applied patch generated 1 new checkstyle issues (total was 471, now 469).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 55s mvn install still works.
          +1 eclipse:eclipse 0m 46s The patch built with eclipse:eclipse.
          +1 findbugs 3m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 42s Pre-build of native portion
          -1 hdfs tests 80m 10s Tests failed in hadoop-hdfs.
              131m 57s  



          Reason Tests
          Timed out tests org.apache.hadoop.hdfs.TestEncryptedTransfer



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751597/HDFS-8865.v3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 22de7c1
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12067/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12067/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12067/testReport/
          Java 1.7.0_55
          uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12067/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 50s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 9m 22s There were no new javac warning messages. +1 javadoc 11m 41s There were no new javadoc warning messages. +1 release audit 0m 27s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 42s The applied patch generated 1 new checkstyle issues (total was 471, now 469). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 55s mvn install still works. +1 eclipse:eclipse 0m 46s The patch built with eclipse:eclipse. +1 findbugs 3m 15s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 42s Pre-build of native portion -1 hdfs tests 80m 10s Tests failed in hadoop-hdfs.     131m 57s   Reason Tests Timed out tests org.apache.hadoop.hdfs.TestEncryptedTransfer Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751597/HDFS-8865.v3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 22de7c1 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12067/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12067/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12067/testReport/ Java 1.7.0_55 uname Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12067/console This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -
          • The checkstyle warning is against DFSConfigKeys.
          • TestEncryptedTransfer failure is not related this patch.
          Show
          kihwal Kihwal Lee added a comment - The checkstyle warning is against DFSConfigKeys. TestEncryptedTransfer failure is not related this patch.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 48s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 2 new or modified test files.
          +1 javac 7m 58s There were no new javac warning messages.
          +1 javadoc 9m 52s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          -1 checkstyle 1m 23s The applied patch generated 1 new checkstyle issues (total was 471, now 469).
          +1 whitespace 0m 1s The patch has no lines that end in whitespace.
          +1 install 1m 27s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 10s Pre-build of native portion
          -1 hdfs tests 188m 5s Tests failed in hadoop-hdfs.
              233m 10s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.TestFSNamesystem
            hadoop.hdfs.server.namenode.TestFileTruncate



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12751597/HDFS-8865.v3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a4d9acc
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12140/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12140/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12140/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12140/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12140/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 48s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 2 new or modified test files. +1 javac 7m 58s There were no new javac warning messages. +1 javadoc 9m 52s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. -1 checkstyle 1m 23s The applied patch generated 1 new checkstyle issues (total was 471, now 469). +1 whitespace 0m 1s The patch has no lines that end in whitespace. +1 install 1m 27s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 2m 29s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 10s Pre-build of native portion -1 hdfs tests 188m 5s Tests failed in hadoop-hdfs.     233m 10s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestFSNamesystem   hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12751597/HDFS-8865.v3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a4d9acc Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12140/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/12140/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12140/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12140/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12140/console This message was automatically generated.
          Hide
          daryn Daryn Sharp added a comment -

          +1 This has made a huge difference, and all the possible style warning were addressed.

          Show
          daryn Daryn Sharp added a comment - +1 This has made a huge difference, and all the possible style warning were addressed.
          Hide
          kihwal Kihwal Lee added a comment -

          Thanks Xiaoyu Yao and Daryn Sharp for reviews. I've committed this to trunk and branch-2.

          Show
          kihwal Kihwal Lee added a comment - Thanks Xiaoyu Yao and Daryn Sharp for reviews. I've committed this to trunk and branch-2.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8364 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8364/)
          HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8364 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8364/ ) HDFS-8865 . Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1050 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1050/)
          HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1050 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1050/ ) HDFS-8865 . Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #317 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/317/)
          HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #317 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/317/ ) HDFS-8865 . Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #322 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/322/)
          HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #322 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/322/ ) HDFS-8865 . Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2266 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2266/)
          HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2266 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2266/ ) HDFS-8865 . Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2247 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2247/)
          HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2247 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2247/ ) HDFS-8865 . Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #309 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/309/)
          HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #309 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/309/ ) HDFS-8865 . Improve quota initialization performance. Contributed by Kihwal Lee. (kihwal: rev b6ceee9bf42eec15891f60a014bbfa47e03f563c) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDiskspaceQuotaUpdate.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaCounts.java
          Hide
          kihwal Kihwal Lee added a comment -

          We are going to apply this to our 2.7 builds. Here is the patch, in case any one is interested.

          Show
          kihwal Kihwal Lee added a comment - We are going to apply this to our 2.7 builds. Here is the patch, in case any one is interested.
          Hide
          nfraison.criteo Nicolas Fraison added a comment -

          We have backported this patch to our 2.6 builds.
          Could you please let me knwo how I can provide it, I don't have any rights to attach the git diff file.

          Show
          nfraison.criteo Nicolas Fraison added a comment - We have backported this patch to our 2.6 builds. Could you please let me knwo how I can provide it, I don't have any rights to attach the git diff file.
          Hide
          kihwal Kihwal Lee added a comment -

          Nicolas Fraison, if you plan to contribute code to the HDFS project in the future, I can add you as a contributor. That will allow you to submit patches. If you want to be a contributor, just let me know.

          As for porting this specific jira to branch-2.6, the release branch for 2.6.5 was just cut, so it will not make the next release. It is also unclear whether 2.6.6 will ever be released.

          Show
          kihwal Kihwal Lee added a comment - Nicolas Fraison , if you plan to contribute code to the HDFS project in the future, I can add you as a contributor. That will allow you to submit patches. If you want to be a contributor, just let me know. As for porting this specific jira to branch-2.6, the release branch for 2.6.5 was just cut, so it will not make the next release. It is also unclear whether 2.6.6 will ever be released.
          Hide
          nfraison.criteo Nicolas Fraison added a comment -

          Kihwal Lee I would appreciate to be added as a contributor.

          Even if it is not pushed to any release the backport will be available here for anyone needing it.

          Show
          nfraison.criteo Nicolas Fraison added a comment - Kihwal Lee I would appreciate to be added as a contributor. Even if it is not pushed to any release the backport will be available here for anyone needing it.
          Hide
          kihwal Kihwal Lee added a comment -

          Just added you as a HDFS contributor.

          Show
          kihwal Kihwal Lee added a comment - Just added you as a HDFS contributor.
          Hide
          nfraison.criteo Nicolas Fraison added a comment -

          Thanks Kihwal Lee
          Patch for 2.6 branch added

          Show
          nfraison.criteo Nicolas Fraison added a comment - Thanks Kihwal Lee Patch for 2.6 branch added

            People

            • Assignee:
              kihwal Kihwal Lee
              Reporter:
              kihwal Kihwal Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development