Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10778

Add -format option to make the output of FileDistribution processor human-readable in OfflineImageViewer

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.7.1
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: tools
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Now It's not directly to understand the output result of the FileDistribution processor that in hdfs oiv command for users. For example, this is a original output:

      Size    NumFiles
      0       22556
      1048576 404971
      2097152 29259
      3145728 16937
      4194304 9197
      5242880 6889
      6291456 4930
      7340032 4070
      8388608 299384
      9437184 274623
      

      Two aspects make that hard to understand for users.

      First, the size column just showed as the number in byte, it's not readable here. The better way is showed with a binary prefix.
      Second, the size column would be better to showed as a size range. It will let users know the value in NumFiles column was counted from A size to B size.

      The expected output result should be this:

      Size Range   NumFiles
      (0 B, 0 B]  1666332
      (0 B, 1 M]        778473
      (1 M, 2 M]      35125
      (2 M, 3 M]      13978
      (3 M, 4 M]      10158
      (4 M, 5 M]      6970
      
      1. HDFS-10778.001.patch
        3 kB
        Yiqun Lin
      2. HDFS-10778.002.patch
        17 kB
        Yiqun Lin
      3. HDFS-10778.003.patch
        17 kB
        Yiqun Lin
      4. HDFS-10778.004.patch
        23 kB
        Yiqun Lin
      5. HDFS-10778.005.patch
        23 kB
        Yiqun Lin
      6. HDFS-10778.006.patch
        23 kB
        Yiqun Lin

        Issue Links

          Activity

          Hide
          linyiqun Yiqun Lin added a comment - - edited

          Some test output result in my test cluster:

          Size Range   NumFiles
          (0 B, 0 B]  16663
          (0 B, 1 M]        778416
          (1 M, 2 M]      35125
          (2 M, 3 M]      13278
          (3 M, 4 M]      10458
          ...
          (126 M, 127 M]  1589
          (127 M, 128 M]  24971
          totalFiles = 960947
          totalDirectories = 30795
          totalBlocks = 996233
          totalSpace = 35208724040079
          maxFileSize = 2593079816
          

          Attach a initial patch.

          Show
          linyiqun Yiqun Lin added a comment - - edited Some test output result in my test cluster: Size Range NumFiles (0 B, 0 B] 16663 (0 B, 1 M] 778416 (1 M, 2 M] 35125 (2 M, 3 M] 13278 (3 M, 4 M] 10458 ... (126 M, 127 M] 1589 (127 M, 128 M] 24971 totalFiles = 960947 totalDirectories = 30795 totalBlocks = 996233 totalSpace = 35208724040079 maxFileSize = 2593079816 Attach a initial patch.
          Hide
          linyiqun Yiqun Lin added a comment -

          Softly ping Akira Ajisaka, I think this is also a improvement for users to use hdfs oiv like HDFS-10691. Could you take a look for this, looking forward to seeing your response.

          Show
          linyiqun Yiqun Lin added a comment - Softly ping Akira Ajisaka , I think this is also a improvement for users to use hdfs oiv like HDFS-10691 . Could you take a look for this, looking forward to seeing your response.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 8m 0s trunk passed
          +1 compile 0m 53s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 3s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 2m 1s trunk passed
          +1 javadoc 1m 3s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 47s the patch passed
          +1 javac 0m 47s the patch passed
          +1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 16 unchanged - 1 fixed = 16 total (was 17)
          +1 mvnsite 0m 55s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 1m 4s the patch passed
          +1 unit 63m 35s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          85m 48s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824495/HDFS-10778.001.patch
          JIRA Issue HDFS-10778
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9157fe755a7f 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / dbcaf99
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16479/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16479/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 0s trunk passed +1 compile 0m 53s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 2m 1s trunk passed +1 javadoc 1m 3s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 47s the patch passed +1 javac 0m 47s the patch passed +1 checkstyle 0m 25s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 16 unchanged - 1 fixed = 16 total (was 17) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 12s the patch passed +1 javadoc 1m 4s the patch passed +1 unit 63m 35s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 85m 48s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824495/HDFS-10778.001.patch JIRA Issue HDFS-10778 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9157fe755a7f 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbcaf99 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16479/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16479/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Yiqun Lin for the patch. The improved output looks great.
          Changing the output format of CLI is incompatible, so would you add a new option to optimize the output? '-h' is good for me. In addition, we need to document the option.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Yiqun Lin for the patch. The improved output looks great. Changing the output format of CLI is incompatible, so would you add a new option to optimize the output? '-h' is good for me. In addition, we need to document the option.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Akira Ajisaka for the quick response. I will attach a new patch to address your comment next week.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Akira Ajisaka for the quick response. I will attach a new patch to address your comment next week.
          Hide
          linyiqun Yiqun Lin added a comment -

          Hi, Akira Ajisaka, now I working on this jira and I found some other problems here.

          so would you add a new option to optimize the output? '-h' is good for me.

          The option '-h' is already used for -help in hdfs oiv command. So it seems we would the other option, now I use a new option -format to instead of that.

          I found another bug when I tested the new option in class OfflineImageViewer, it is missing the following code in method OfflineImageViewer#buildOptions:

          options.addOption("maxSize", true, "");
           options.addOption("step", true, "");
          

          Then leads the ParseException being threw when doing the parser.parse(options, args).

          org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: -maxSize
          

          The test output in my local env:

          with -format option:
          Size Range	NumFiles
          (0 B, 8 B]	12
          totalFiles = 12
          totalDirectories = 9
          totalBlocks = 12
          totalSpace = 12
          maxFileSize = 1
          
          without -format option:
          Size	NumFiles
          8	12
          totalFiles = 12
          totalDirectories = 9
          totalBlocks = 12
          totalSpace = 12
          maxFileSize = 1
          

          Finally, attach a new patch for this, thanks for the review.

          Show
          linyiqun Yiqun Lin added a comment - Hi, Akira Ajisaka , now I working on this jira and I found some other problems here. so would you add a new option to optimize the output? '-h' is good for me. The option '-h' is already used for -help in hdfs oiv command. So it seems we would the other option, now I use a new option -format to instead of that. I found another bug when I tested the new option in class OfflineImageViewer , it is missing the following code in method OfflineImageViewer#buildOptions : options.addOption( "maxSize" , true , ""); options.addOption( "step" , true , ""); Then leads the ParseException being threw when doing the parser.parse(options, args) . org.apache.commons.cli.UnrecognizedOptionException: Unrecognized option: -maxSize The test output in my local env: with -format option: Size Range NumFiles (0 B, 8 B] 12 totalFiles = 12 totalDirectories = 9 totalBlocks = 12 totalSpace = 12 maxFileSize = 1 without -format option: Size NumFiles 8 12 totalFiles = 12 totalDirectories = 9 totalBlocks = 12 totalSpace = 12 maxFileSize = 1 Finally, attach a new patch for this, thanks for the review.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Yiqun Lin for updating the patch!

          The option '-h' is already used for -help in hdfs oiv command. So it seems we would the other option, now I use a new option -format to instead of that.

          Agreed.

          I found another bug when I tested the new option in class OfflineImageViewer, it is missing the following code in method OfflineImageViewer#buildOptions:

          Nice catch. Would you file another jira for this issue and split the patch? The bug is similar to HDFS-5866.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Yiqun Lin for updating the patch! The option '-h' is already used for -help in hdfs oiv command. So it seems we would the other option, now I use a new option -format to instead of that. Agreed. I found another bug when I tested the new option in class OfflineImageViewer, it is missing the following code in method OfflineImageViewer#buildOptions: Nice catch. Would you file another jira for this issue and split the patch? The bug is similar to HDFS-5866 .
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 47s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 40s trunk passed
          +1 javadoc 0m 53s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 41s the patch passed
          +1 javac 0m 41s the patch passed
          -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 120 unchanged - 36 fixed = 122 total (was 156)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 52s the patch passed
          +1 unit 59m 40s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          78m 26s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824761/HDFS-10778.002.patch
          JIRA Issue HDFS-10778
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ac792a567872 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 115ecb5
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16493/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16493/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16493/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 47s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 40s trunk passed +1 javadoc 0m 53s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 41s the patch passed +1 javac 0m 41s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 120 unchanged - 36 fixed = 122 total (was 156) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 52s the patch passed +1 unit 59m 40s hadoop-hdfs in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 78m 26s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824761/HDFS-10778.002.patch JIRA Issue HDFS-10778 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ac792a567872 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 115ecb5 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16493/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16493/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16493/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Akira Ajisaka for the review.

          Would you file another jira for this issue and split the patch?

          Done. I created the HDFS-10783 for tracking this and posted a simple patch for that.

          Update the patch of this jira.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Akira Ajisaka for the review. Would you file another jira for this issue and split the patch? Done. I created the HDFS-10783 for tracking this and posted a simple patch for that. Update the patch of this jira.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 9s trunk passed
          +1 compile 0m 55s trunk passed
          +1 checkstyle 0m 30s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 59s trunk passed
          +1 mvninstall 0m 55s the patch passed
          +1 compile 0m 53s the patch passed
          +1 javac 0m 53s the patch passed
          -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 120 unchanged - 36 fixed = 122 total (was 156)
          +1 mvnsite 0m 59s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 7s the patch passed
          +1 javadoc 0m 59s the patch passed
          -1 unit 64m 28s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          86m 39s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestEncryptionZones
            hadoop.hdfs.TestBlockStoragePolicy



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824789/HDFS-10778.003.patch
          JIRA Issue HDFS-10778
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3453b4065648 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 115ecb5
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16495/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16495/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16495/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 9s trunk passed +1 compile 0m 55s trunk passed +1 checkstyle 0m 30s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 59s trunk passed +1 mvninstall 0m 55s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed -0 checkstyle 0m 30s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 120 unchanged - 36 fixed = 122 total (was 156) +1 mvnsite 0m 59s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 59s the patch passed -1 unit 64m 28s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 86m 39s Reason Tests Failed junit tests hadoop.hdfs.TestEncryptionZones   hadoop.hdfs.TestBlockStoragePolicy Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824789/HDFS-10778.003.patch JIRA Issue HDFS-10778 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3453b4065648 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 115ecb5 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16495/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16495/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16495/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16495/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          Attach a new patch to fix checkstyle warnings.

          Show
          linyiqun Yiqun Lin added a comment - Attach a new patch to fix checkstyle warnings.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 45s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 0m 52s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 46s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 61 unchanged - 94 fixed = 79 total (was 155)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 53s the patch passed
          +1 unit 58m 0s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          76m 55s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824818/HDFS-10778.004.patch
          JIRA Issue HDFS-10778
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 80f2cd6b9572 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 115ecb5
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16497/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16497/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16497/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 45s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 46s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 61 unchanged - 94 fixed = 79 total (was 155) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 53s the patch passed +1 unit 58m 0s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 76m 55s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12824818/HDFS-10778.004.patch JIRA Issue HDFS-10778 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 80f2cd6b9572 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 115ecb5 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16497/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16497/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16497/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          linyiqun Yiqun Lin added a comment -

          Hi Akira Ajisaka, can we go ahead for this JIRA, the latest patch can satisfy with you? Thanks a lot if you have time to take a look for this.

          Show
          linyiqun Yiqun Lin added a comment - Hi Akira Ajisaka , can we go ahead for this JIRA, the latest patch can satisfy with you? Thanks a lot if you have time to take a look for this.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          The patch looks good to me. Would you rebase the patch?
          Sorry for the late response.

          Show
          ajisakaa Akira Ajisaka added a comment - The patch looks good to me. Would you rebase the patch? Sorry for the late response.
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Akira Ajisaka for the review! Attach a new patch to rebase the current code, pending jenkins.

          Show
          linyiqun Yiqun Lin added a comment - Thanks Akira Ajisaka for the review! Attach a new patch to rebase the current code, pending jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 44s trunk passed
          +1 compile 0m 47s trunk passed
          +1 checkstyle 0m 31s trunk passed
          +1 mvnsite 0m 56s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 45s trunk passed
          +1 javadoc 1m 3s trunk passed
          +1 mvninstall 0m 51s the patch passed
          +1 compile 0m 46s the patch passed
          +1 javac 0m 46s the patch passed
          -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 61 unchanged - 94 fixed = 79 total (was 155)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 76m 21s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          97m 8s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10778
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827039/HDFS-10778.005.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ccf1e568ed17 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 07650bc
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16633/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16633/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16633/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 44s trunk passed +1 compile 0m 47s trunk passed +1 checkstyle 0m 31s trunk passed +1 mvnsite 0m 56s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 45s trunk passed +1 javadoc 1m 3s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 46s the patch passed +1 javac 0m 46s the patch passed -0 checkstyle 0m 26s hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 61 unchanged - 94 fixed = 79 total (was 155) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 76m 21s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 97m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10778 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827039/HDFS-10778.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ccf1e568ed17 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 07650bc Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16633/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16633/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16633/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Yiqun Lin for updating the patch. I tried your patch and get the following output:

          [centos@ip-172-31-21-203 conf]$ hdfs oiv -p FileDistribution -format -step 30 -maxSize 300 -i /hadoop/dfs/name/current/fsimage_0000000000000000307
          Processed 0 inodes.
          Size Range	NumFiles
          (0 B, 30 B]	2
          (270 B, 300 B]	32
          totalFiles = 34
          totalDirectories = 13
          totalBlocks = 34
          totalSpace = 288198
          maxFileSize = 160321
          

          Actually maxFileSize is 160321 but the output says the file is in (270 B, 300 B]. Would you fix it to output (270B, maxFileSize]?

          If a fsimage includes empty files, the output is as follows:

          Size Range	NumFiles
          (0 B, 0 B]	1
          (0 B, 30 B]	2
          

          I'm thinking [0 B, 0 B] is better than (0 B, 0 B].

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Yiqun Lin for updating the patch. I tried your patch and get the following output: [centos@ip-172-31-21-203 conf]$ hdfs oiv -p FileDistribution -format -step 30 -maxSize 300 -i /hadoop/dfs/name/current/fsimage_0000000000000000307 Processed 0 inodes. Size Range NumFiles (0 B, 30 B] 2 (270 B, 300 B] 32 totalFiles = 34 totalDirectories = 13 totalBlocks = 34 totalSpace = 288198 maxFileSize = 160321 Actually maxFileSize is 160321 but the output says the file is in (270 B, 300 B] . Would you fix it to output (270B, maxFileSize]? If a fsimage includes empty files, the output is as follows: Size Range NumFiles (0 B, 0 B] 1 (0 B, 30 B] 2 I'm thinking [0 B, 0 B] is better than (0 B, 0 B].
          Hide
          linyiqun Yiqun Lin added a comment -

          Thanks Akira Ajisaka for the review. Attach a new patch to make a minor change. The test output in my local:

          Size Range	NumFiles
          [0 B, 0 B]	4
          (0 B, 4 B]	1
          (8 B, 12 B]	1
          (12 B, 16 B]	1
          (16 B, 21 B]	1
          totalFiles = 8
          totalDirectories = 2
          totalBlocks = 4
          totalSpace = 48
          maxFileSize = 21
          
          Show
          linyiqun Yiqun Lin added a comment - Thanks Akira Ajisaka for the review. Attach a new patch to make a minor change. The test output in my local: Size Range NumFiles [0 B, 0 B] 4 (0 B, 4 B] 1 (8 B, 12 B] 1 (12 B, 16 B] 1 (16 B, 21 B] 1 totalFiles = 8 totalDirectories = 2 totalBlocks = 4 totalSpace = 48 maxFileSize = 21
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 31s trunk passed
          +1 compile 0m 43s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 54s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 61 unchanged - 94 fixed = 79 total (was 155)
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 45s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 57m 28s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          77m 43s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10778
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827126/HDFS-10778.006.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux df13f6a83387 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 07650bc
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16640/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16640/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16640/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16640/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 31s trunk passed +1 compile 0m 43s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 61 unchanged - 94 fixed = 79 total (was 155) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 45s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 57m 28s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 77m 43s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestEditLogTailer Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10778 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12827126/HDFS-10778.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux df13f6a83387 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 07650bc Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16640/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16640/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16640/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16640/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1, I verified the option in both oiv and oiv_legacy commands. I'll commit this tomorrow if there are no objections.

          Show
          ajisakaa Akira Ajisaka added a comment - +1, I verified the option in both oiv and oiv_legacy commands. I'll commit this tomorrow if there are no objections.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk and branch-2. Thanks Yiqun Lin for the contribution!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk and branch-2. Thanks Yiqun Lin for the contribution!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10407 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10407/)
          HDFS-10778. Add -format option to make the output of FileDistribution (aajisaka: rev 63f594892ecd4687e37a99790288e36eb278849f)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionVisitor.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10407 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10407/ ) HDFS-10778 . Add -format option to make the output of FileDistribution (aajisaka: rev 63f594892ecd4687e37a99790288e36eb278849f) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionVisitor.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsImageViewer.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
          Hide
          linyiqun Yiqun Lin added a comment - - edited

          Thanks for the commit, Akira Ajisaka. Now I looked into the document for the OfflineImageVer, I found something still not completed. For example, the option maxSize, step, format were not added in hdfs oiv_legacy. But actually they are making sence. I filed a new JIRA HDFS-10847 to make a quick fix.

          Show
          linyiqun Yiqun Lin added a comment - - edited Thanks for the commit, Akira Ajisaka . Now I looked into the document for the OfflineImageVer, I found something still not completed. For example, the option maxSize , step , format were not added in hdfs oiv_legacy. But actually they are making sence. I filed a new JIRA HDFS-10847 to make a quick fix.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I forgot to add "Contributed by Yiqun Lin." to the commit log. I'm sorry for that.

          I filed a new JIRA HDFS-10847 to make a quick fix.

          Thank you for the follow up!

          Show
          ajisakaa Akira Ajisaka added a comment - I forgot to add "Contributed by Yiqun Lin." to the commit log. I'm sorry for that. I filed a new JIRA HDFS-10847 to make a quick fix. Thank you for the follow up!

            People

            • Assignee:
              linyiqun Yiqun Lin
              Reporter:
              linyiqun Yiqun Lin
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development