Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8581

ContentSummary on / skips further counts on yielding lock

    Details

    • Hadoop Flags:
      Reviewed

      Description

      If one directory such as "/result" exists about 200000 files, then when execute "hdfs dfs -count /", the result will go wrong. For all directories whose name after "/result", file num will not be included.

      My cluster see as below, "/result_1433858936" is the directory exist huge files, and files in "/sparkJobHistory", "/tmp", "/user" are not included

      vm-221:/export1/BigData/current # hdfs dfs -ls /
      15/06/11 11:00:17 INFO hdfs.PeerCache: SocketCache disabled.
      Found 9 items
      rw-rr- 3 hdfs supergroup 0 2015-06-08 12:10 /PRE_CREATE_DIR.SUCCESS
      drwxr-x--- - flume hadoop 0 2015-06-08 12:08 /flume
      drwx------ - hbase hadoop 0 2015-06-10 15:25 /hbase
      drwxr-xr-x - hdfs supergroup 0 2015-06-10 17:19 /hyt
      drwxrwxrwx - mapred hadoop 0 2015-06-08 12:08 /mr-history
      drwxr-xr-x - hdfs supergroup 0 2015-06-09 22:10 /result_1433858936
      drwxrwxrwx - spark supergroup 0 2015-06-10 19:15 /sparkJobHistory
      drwxrwxrwx - hdfs hadoop 0 2015-06-08 12:14 /tmp
      drwxrwxrwx - hdfs hadoop 0 2015-06-09 21:57 /user
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /
      15/06/11 11:00:24 INFO hdfs.PeerCache: SocketCache disabled.
      1043 171536 1756375688 /
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /PRE_CREATE_DIR.SUCCESS
      15/06/11 11:00:30 INFO hdfs.PeerCache: SocketCache disabled.
      0 1 0 /PRE_CREATE_DIR.SUCCESS
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /flume
      15/06/11 11:00:41 INFO hdfs.PeerCache: SocketCache disabled.
      1 0 0 /flume
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /hbase
      15/06/11 11:00:49 INFO hdfs.PeerCache: SocketCache disabled.
      36 18 14807 /hbase
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /hyt
      15/06/11 11:01:09 INFO hdfs.PeerCache: SocketCache disabled.
      1 0 0 /hyt
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /mr-history
      15/06/11 11:01:18 INFO hdfs.PeerCache: SocketCache disabled.
      3 0 0 /mr-history
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /result_1433858936
      15/06/11 11:01:29 INFO hdfs.PeerCache: SocketCache disabled.
      1001 171517 1756360881 /result_1433858936
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /sparkJobHistory
      15/06/11 11:01:41 INFO hdfs.PeerCache: SocketCache disabled.
      1 3 21785 /sparkJobHistory
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /tmp
      15/06/11 11:01:48 INFO hdfs.PeerCache: SocketCache disabled.
      17 6 35958 /tmp
      vm-221:/export1/BigData/current #
      vm-221:/export1/BigData/current # hdfs dfs -count /user
      15/06/11 11:01:55 INFO hdfs.PeerCache: SocketCache disabled.
      12 1 19077 /user

      1. HDFS-8581.1.patch
        3 kB
        J.Andreina
      2. HDFS-8581.2.patch
        3 kB
        J.Andreina
      3. HDFS-8581.3.patch
        3 kB
        J.Andreina
      4. HDFS-8581.4.patch
        3 kB
        J.Andreina

        Issue Links

          Activity

          Hide
          andreina J.Andreina added a comment -

          Issues in count is :

          While traversing throught subFolders for calculating the file count , if the value execeeds "dfs.content-summary.limit" ,
          then traversing through other child nodes for calculating the file count is been skipped.

          Scenario :
          =========
          dfs.content-summary.limit = 5000
          /Folder1   - 10 file
          /Folder2   - 6000 files
          /Folder3   - 10 file
          
          Now when i do count on "/"  it returns me file count as 6010 files , instead of 6020 files.. 
          Files under Folder3 is not counted.
          

          Attached an initial patch.
          Please review.

          Show
          andreina J.Andreina added a comment - Issues in count is : While traversing throught subFolders for calculating the file count , if the value execeeds "dfs.content-summary.limit" , then traversing through other child nodes for calculating the file count is been skipped. Scenario : ========= dfs.content-summary.limit = 5000 /Folder1 - 10 file /Folder2 - 6000 files /Folder3 - 10 file Now when i do count on "/" it returns me file count as 6010 files , instead of 6020 files.. Files under Folder3 is not counted. Attached an initial patch. Please review.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 18m 4s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 39s There were no new javac warning messages.
          +1 javadoc 9m 34s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 2m 8s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 33s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 3m 16s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 16s Pre-build of native portion
          -1 hdfs tests 154m 31s Tests failed in hadoop-hdfs.
              201m 0s  



          Reason Tests
          Failed unit tests hadoop.hdfs.TestRead
            hadoop.hdfs.TestClientReportBadBlock
            hadoop.hdfs.TestAppendSnapshotTruncate
            hadoop.hdfs.TestFileAppendRestart
            hadoop.TestRefreshCallQueue
            hadoop.hdfs.TestListFilesInDFS
            hadoop.security.TestPermissionSymlinks
            hadoop.hdfs.TestDFSRollback
            hadoop.hdfs.TestFileConcurrentReader
            hadoop.hdfs.TestFileAppend2
            hadoop.hdfs.TestDFSClientRetries
            hadoop.hdfs.TestGetFileChecksum
            hadoop.hdfs.crypto.TestHdfsCryptoStreams
            hadoop.hdfs.TestBlockStoragePolicy
            hadoop.hdfs.TestCrcCorruption
            hadoop.hdfs.TestDFSUpgrade
            hadoop.hdfs.TestLeaseRecovery2
            hadoop.cli.TestXAttrCLI
            hadoop.hdfs.TestDFSShell
            hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer
            hadoop.security.TestPermission
            hadoop.hdfs.TestFileCreationDelete
            hadoop.hdfs.TestDFSStorageStateRecovery
            hadoop.hdfs.TestAppendDifferentChecksum
            hadoop.hdfs.TestRemoteBlockReader
            hadoop.hdfs.TestRestartDFS
            hadoop.cli.TestHDFSCLI
            hadoop.hdfs.TestHDFSFileSystemContract
            hadoop.cli.TestAclCLI
            hadoop.hdfs.TestDFSPermission
            hadoop.cli.TestCryptoAdminCLI
            hadoop.hdfs.TestRollingUpgrade
            hadoop.hdfs.TestDFSStartupVersions
            hadoop.hdfs.TestWriteBlockGetsBlockLengthHint
            hadoop.hdfs.TestAbandonBlock
            hadoop.hdfs.TestFetchImage
            hadoop.hdfs.TestDatanodeDeath
            hadoop.hdfs.TestDFSClientExcludedNodes
            hadoop.hdfs.TestDFSOutputStream
            hadoop.hdfs.TestReservedRawPaths



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739214/HDFS-8581.1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / d8dcfa9
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11319/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11319/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11319/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 18m 4s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 39s There were no new javac warning messages. +1 javadoc 9m 34s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 2m 8s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 33s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 3m 16s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 16s Pre-build of native portion -1 hdfs tests 154m 31s Tests failed in hadoop-hdfs.     201m 0s   Reason Tests Failed unit tests hadoop.hdfs.TestRead   hadoop.hdfs.TestClientReportBadBlock   hadoop.hdfs.TestAppendSnapshotTruncate   hadoop.hdfs.TestFileAppendRestart   hadoop.TestRefreshCallQueue   hadoop.hdfs.TestListFilesInDFS   hadoop.security.TestPermissionSymlinks   hadoop.hdfs.TestDFSRollback   hadoop.hdfs.TestFileConcurrentReader   hadoop.hdfs.TestFileAppend2   hadoop.hdfs.TestDFSClientRetries   hadoop.hdfs.TestGetFileChecksum   hadoop.hdfs.crypto.TestHdfsCryptoStreams   hadoop.hdfs.TestBlockStoragePolicy   hadoop.hdfs.TestCrcCorruption   hadoop.hdfs.TestDFSUpgrade   hadoop.hdfs.TestLeaseRecovery2   hadoop.cli.TestXAttrCLI   hadoop.hdfs.TestDFSShell   hadoop.hdfs.protocol.datatransfer.sasl.TestSaslDataTransfer   hadoop.security.TestPermission   hadoop.hdfs.TestFileCreationDelete   hadoop.hdfs.TestDFSStorageStateRecovery   hadoop.hdfs.TestAppendDifferentChecksum   hadoop.hdfs.TestRemoteBlockReader   hadoop.hdfs.TestRestartDFS   hadoop.cli.TestHDFSCLI   hadoop.hdfs.TestHDFSFileSystemContract   hadoop.cli.TestAclCLI   hadoop.hdfs.TestDFSPermission   hadoop.cli.TestCryptoAdminCLI   hadoop.hdfs.TestRollingUpgrade   hadoop.hdfs.TestDFSStartupVersions   hadoop.hdfs.TestWriteBlockGetsBlockLengthHint   hadoop.hdfs.TestAbandonBlock   hadoop.hdfs.TestFetchImage   hadoop.hdfs.TestDatanodeDeath   hadoop.hdfs.TestDFSClientExcludedNodes   hadoop.hdfs.TestDFSOutputStream   hadoop.hdfs.TestReservedRawPaths Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739214/HDFS-8581.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / d8dcfa9 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/11319/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/11319/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11319/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 patch 0m 0s The patch command could not apply the patch during dryrun.



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739214/HDFS-8581.1.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 50887e5
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11864/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 patch 0m 0s The patch command could not apply the patch during dryrun. Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739214/HDFS-8581.1.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 50887e5 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/11864/console This message was automatically generated.
          Hide
          andreina J.Andreina added a comment -

          Patch is updated. Please review.

          Show
          andreina J.Andreina added a comment - Patch is updated. Please review.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 43s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 52s There were no new javac warning messages.
          +1 javadoc 10m 0s There were no new javadoc warning messages.
          +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 24s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 14s Pre-build of native portion
          +1 hdfs tests 162m 9s Tests passed in hadoop-hdfs.
              207m 24s  



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752211/HDFS-8581.2.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / eee0d45
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/whitespace.txt
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12115/testReport/
          Java 1.7.0_55
          uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12115/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 43s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 52s There were no new javac warning messages. +1 javadoc 10m 0s There were no new javadoc warning messages. +1 release audit 0m 24s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 24s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 31s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 14s Pre-build of native portion +1 hdfs tests 162m 9s Tests passed in hadoop-hdfs.     207m 24s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752211/HDFS-8581.2.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / eee0d45 Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/whitespace.txt hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12115/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12115/testReport/ Java 1.7.0_55 uname Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12115/console This message was automatically generated.
          Hide
          andreina J.Andreina added a comment -

          Updated the patch.
          Please review.

          Show
          andreina J.Andreina added a comment - Updated the patch. Please review.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 17m 23s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 42s There were no new javac warning messages.
          +1 javadoc 9m 57s There were no new javadoc warning messages.
          +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 55s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 37s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 28s Pre-build of native portion
          -1 hdfs tests 188m 56s Tests failed in hadoop-hdfs.
              233m 27s  



          Reason Tests
          Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752391/HDFS-8581.3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / a4d9acc
          Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12129/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12129/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12129/testReport/
          Java 1.7.0_55
          uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12129/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 17m 23s Pre-patch trunk has 4 extant Findbugs (version 3.0.0) warnings. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 9m 57s There were no new javadoc warning messages. +1 release audit 0m 25s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 55s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 37s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 28s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 28s Pre-build of native portion -1 hdfs tests 188m 56s Tests failed in hadoop-hdfs.     233m 27s   Reason Tests Failed unit tests hadoop.hdfs.server.namenode.TestFileTruncate Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752391/HDFS-8581.3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / a4d9acc Pre-patch Findbugs warnings https://builds.apache.org/job/PreCommit-HDFS-Build/12129/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12129/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12129/testReport/ Java 1.7.0_55 uname Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12129/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 47s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 8m 0s There were no new javac warning messages.
          +1 javadoc 10m 10s There were no new javadoc warning messages.
          +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 24s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 30s mvn install still works.
          +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse.
          +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 11s Pre-build of native portion
          -1 hdfs tests 92m 27s Tests failed in hadoop-hdfs.
              138m 0s  



          Reason Tests
          Failed unit tests hadoop.hdfs.web.TestWebHDFSOAuth2
            hadoop.hdfs.qjournal.client.TestQJMWithFaults
            hadoop.hdfs.security.TestDelegationToken



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12752391/HDFS-8581.3.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / b469ac5
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12277/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12277/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12277/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 47s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 8m 0s There were no new javac warning messages. +1 javadoc 10m 10s There were no new javadoc warning messages. +1 release audit 0m 22s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 24s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 30s mvn install still works. +1 eclipse:eclipse 0m 33s The patch built with eclipse:eclipse. +1 findbugs 2m 32s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 11s Pre-build of native portion -1 hdfs tests 92m 27s Tests failed in hadoop-hdfs.     138m 0s   Reason Tests Failed unit tests hadoop.hdfs.web.TestWebHDFSOAuth2   hadoop.hdfs.qjournal.client.TestQJMWithFaults   hadoop.hdfs.security.TestDelegationToken Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12752391/HDFS-8581.3.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / b469ac5 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12277/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12277/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12277/console This message was automatically generated.
          Hide
          andreina J.Andreina added a comment -

          Test Case failures are not related to this patch
          Please review

          Show
          andreina J.Andreina added a comment - Test Case failures are not related to this patch Please review
          Hide
          vinayrpet Vinayakumar B added a comment -

          While traversing throught subFolders for calculating the file count , if the value execeeds "dfs.content-summary.limit" , then traversing through other child nodes for calculating the file count is been skipped.

          Thanks J.Andreina for the analysis and the patch.

          Your patch is almost correct.
          I observed that this problem will come only with "/". Not with any low level directories. It will not skip any contents for low-level dirs even with any number of files.

          Reason is, "/" always have parent as null. So skipping further count, treating that directory was deleted in between when the lock was released.

          Correct fix would be as below.

                // The locks were released and reacquired. Check parent first.
                if (!isRoot() && getParent() == null) {
                  // Stop further counting and return whatever we have so far.
                  break;
                }

          Also In Test, you make it run faster by skipping the start of datanodes as you are not really worried about the content of files.
          You can start 0 datanodes and you can create empty files. This will make test run little faster.

          Show
          vinayrpet Vinayakumar B added a comment - While traversing throught subFolders for calculating the file count , if the value execeeds "dfs.content-summary.limit" , then traversing through other child nodes for calculating the file count is been skipped. Thanks J.Andreina for the analysis and the patch. Your patch is almost correct. I observed that this problem will come only with "/". Not with any low level directories. It will not skip any contents for low-level dirs even with any number of files. Reason is, "/" always have parent as null. So skipping further count, treating that directory was deleted in between when the lock was released. Correct fix would be as below. // The locks were released and reacquired. Check parent first. if (!isRoot() && getParent() == null ) { // Stop further counting and return whatever we have so far. break ; } Also In Test, you make it run faster by skipping the start of datanodes as you are not really worried about the content of files. You can start 0 datanodes and you can create empty files. This will make test run little faster.
          Hide
          andreina J.Andreina added a comment -

          Thanks Vinayakumar B for the review comments.
          Updated the patch . please review.

          Show
          andreina J.Andreina added a comment - Thanks Vinayakumar B for the review comments. Updated the patch . please review.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 17m 26s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 42s There were no new javac warning messages.
          +1 javadoc 9m 59s There were no new javadoc warning messages.
          +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 1m 18s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 26s mvn install still works.
          +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse.
          +1 findbugs 2m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          +1 native 3m 7s Pre-build of native portion
          -1 hdfs tests 169m 4s Tests failed in hadoop-hdfs.
              213m 27s  



          Reason Tests
          Failed unit tests hadoop.hdfs.web.TestWebHDFSOAuth2



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12754899/HDFS-8581.4.patch
          Optional Tests javadoc javac unit findbugs checkstyle
          git revision trunk / 0113e45
          hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12354/artifact/patchprocess/testrun_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12354/testReport/
          Java 1.7.0_55
          uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12354/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 17m 26s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 42s There were no new javac warning messages. +1 javadoc 9m 59s There were no new javadoc warning messages. +1 release audit 0m 23s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 1m 18s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 26s mvn install still works. +1 eclipse:eclipse 0m 34s The patch built with eclipse:eclipse. +1 findbugs 2m 25s The patch does not introduce any new Findbugs (version 3.0.0) warnings. +1 native 3m 7s Pre-build of native portion -1 hdfs tests 169m 4s Tests failed in hadoop-hdfs.     213m 27s   Reason Tests Failed unit tests hadoop.hdfs.web.TestWebHDFSOAuth2 Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12754899/HDFS-8581.4.patch Optional Tests javadoc javac unit findbugs checkstyle git revision trunk / 0113e45 hadoop-hdfs test log https://builds.apache.org/job/PreCommit-HDFS-Build/12354/artifact/patchprocess/testrun_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/12354/testReport/ Java 1.7.0_55 uname Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-HDFS-Build/12354/console This message was automatically generated.
          Hide
          vinayrpet Vinayakumar B added a comment -

          +1 for the latest patch.

          Test failure is unrelated

          Show
          vinayrpet Vinayakumar B added a comment - +1 for the latest patch. Test failure is unrelated
          Hide
          vinayrpet Vinayakumar B added a comment -

          Committed to trunk and branch-2.
          thanks for the contribution J.Andreina.

          Show
          vinayrpet Vinayakumar B added a comment - Committed to trunk and branch-2. thanks for the contribution J.Andreina .
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8423 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8423/)
          HDFS-8581. ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8423 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8423/ ) HDFS-8581 . ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1101 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1101/)
          HDFS-8581. ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1101 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1101/ ) HDFS-8581 . ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #370 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/370/)
          HDFS-8581. ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7)

          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #370 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/370/ ) HDFS-8581 . ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7) hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #363 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/363/)
          HDFS-8581. ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #363 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/363/ ) HDFS-8581 . ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2289 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2289/)
          HDFS-8581. ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7)

          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2289 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2289/ ) HDFS-8581 . ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #350 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/350/)
          HDFS-8581. ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #350 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/350/ ) HDFS-8581 . ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2312 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2312/)
          HDFS-8581. ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7)

          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
          • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2312 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2312/ ) HDFS-8581 . ContentSummary on / skips further counts on yielding lock (contributed by J.Andreina) (vinayakumarb: rev 4014ce5990bff9b0ecb3d38a633d40eaf6cf07a7) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          Hide
          kihwal Kihwal Lee added a comment -

          Cherry-picked to branch-2.7 and branch-2.6. Thanks again for reporting and fixing the issue.

          Show
          kihwal Kihwal Lee added a comment - Cherry-picked to branch-2.7 and branch-2.6. Thanks again for reporting and fixing the issue.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              andreina J.Andreina
              Reporter:
              tongshiquan tongshiquan
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development