Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: documentation
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      OfflineImageViewer is renewed to handle the new format of fsimage by HDFS-5698 (fsimage in protobuf).

      We should document the followings:

      • The tool can handle the layout version of Hadoop 2.4 and up. (If you want to handle the older version, you can use OfflineImageViewer of Hadoop 2.3)
      • Delimited, Indented, and Ls processor were removed.
      • A new Web processor, which supersedes the Ls processor, was added.
      1. HDFS-5865.2.patch
        23 kB
        Akira AJISAKA
      2. HDFS-5865.patch
        18 kB
        Akira AJISAKA

        Activity

        Karthik Kambatla (Inactive) made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1769 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1769/)
        HDFS-5865. Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1769 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1769/ ) HDFS-5865 . Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1743 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1743/)
        HDFS-5865. Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1743 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1743/ ) HDFS-5865 . Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #552 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/552/)
        HDFS-5865. Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #552 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/552/ ) HDFS-5865 . Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #5574 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5574/)
        HDFS-5865. Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5574 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5574/ ) HDFS-5865 . Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Haohui Mai made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 2.5.0 [ 12326264 ]
        Resolution Fixed [ 1 ]
        Hide
        Haohui Mai added a comment -

        I've committed the patch to trunk and branch-2. Thanks Akira AJISAKA for the contribution.

        Show
        Haohui Mai added a comment - I've committed the patch to trunk and branch-2. Thanks Akira AJISAKA for the contribution.
        Hide
        Haohui Mai added a comment -

        +1

        Show
        Haohui Mai added a comment - +1
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12641879/HDFS-5865.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6731//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6731//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641879/HDFS-5865.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6731//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6731//console This message is automatically generated.
        Akira AJISAKA made changes -
        Description OfflineImageViewer is renewed to handle the new format of fsimage by HDFS-5698 (fsimage in protobuf).

        We should document followings:

        * The tool can handle the layout version of Hadoop 2.4 and up. (If you want to handle the older version, you can use OfflineImageViewer of Hadoop 2.3)
        * Remove deprecated options such as Delimited and Indented processor.
        OfflineImageViewer is renewed to handle the new format of fsimage by HDFS-5698 (fsimage in protobuf).

        We should document the followings:

        * The tool can handle the layout version of Hadoop 2.4 and up. (If you want to handle the older version, you can use OfflineImageViewer of Hadoop 2.3)
        * Delimited, Indented, and Ls processor were removed.
        * A new Web processor, which supersedes the Ls processor, was added.
        Akira AJISAKA made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Akira AJISAKA made changes -
        Attachment HDFS-5865.2.patch [ 12641879 ]
        Hide
        Akira AJISAKA added a comment -

        Attaching a patch to add the description of the Web processor.

        Show
        Akira AJISAKA added a comment - Attaching a patch to add the description of the Web processor.
        Hide
        Akira AJISAKA added a comment -

        Thanks for the review, Haohui.

        why using \u007c instead of |, which is in the old document?

        I thought | was not available in a cell, but I confirmed it was available. I'll use | in the next patch.

        There are several paragraphs that are specific to the ls output processor haven't been cleaned up yet.

        Ls was not cleaned up yet in the patch because this patch was for branch-2.4. I'll update the patch.

        Show
        Akira AJISAKA added a comment - Thanks for the review, Haohui. why using \u007c instead of |, which is in the old document? I thought | was not available in a cell, but I confirmed it was available. I'll use | in the next patch. There are several paragraphs that are specific to the ls output processor haven't been cleaned up yet. Ls was not cleaned up yet in the patch because this patch was for branch-2.4. I'll update the patch.
        Hide
        Haohui Mai added a comment - - edited
        +| <<<-i>>>\u007c<<<--inputFile>>> <input file> | Specify the input fsimage file
        +| <<<-o>>>\u007c<<<--outputFile>>> <output file> | Specify the output filename,
        +| <<<-p>>>\u007c<<<--processor>>> <processor> | Specify the image processor to
        +| <<<-h>>>\u007c<<<--help>>>| Display the tool usage and help information and
        ...
        

        why using \u007c instead of |, which is in the old document?

        There are several paragraphs that are specific to the ls output processor haven't been cleaned up yet. For example:

           This will create a file named fsimage.txt in the current directory
           using the Ls output processor. For very large image files, this process
           may take several minutes.
        
           One can specify which output processor via the command-line switch -p.
           For instance:
        
        Show
        Haohui Mai added a comment - - edited +| <<<-i>>>\u007c<<<--inputFile>>> <input file> | Specify the input fsimage file +| <<<-o>>>\u007c<<<--outputFile>>> <output file> | Specify the output filename, +| <<<-p>>>\u007c<<<--processor>>> <processor> | Specify the image processor to +| <<<-h>>>\u007c<<<--help>>>| Display the tool usage and help information and ... why using \u007c instead of | , which is in the old document? There are several paragraphs that are specific to the ls output processor haven't been cleaned up yet. For example: This will create a file named fsimage.txt in the current directory using the Ls output processor. For very large image files, this process may take several minutes. One can specify which output processor via the command-line switch -p. For instance:
        Akira AJISAKA made changes -
        Summary Update OfflineImageViewer document for branch-2.4 Update OfflineImageViewer document
        Akira AJISAKA made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Akira AJISAKA added a comment -

        Since 2.4.0-rc0 will come soon, I'll update the description and create a patch for branch-2.

        Show
        Akira AJISAKA added a comment - Since 2.4.0-rc0 will come soon, I'll update the description and create a patch for branch-2.
        Hide
        Akira AJISAKA added a comment -

        I can provide a patch of branch-2, but I think 2.4 is the first. We should document the deprecated options and backward incompatibility because major changes have been committed in 2.4 branch also.

        Show
        Akira AJISAKA added a comment - I can provide a patch of branch-2, but I think 2.4 is the first. We should document the deprecated options and backward incompatibility because major changes have been committed in 2.4 branch also.
        Hide
        Haohui Mai added a comment -

        Downgrading to minor since documentation issues block no functionalities.

        By the way, since several major changes (e.g., HDFS-5978 and HDFS-6164) have been committed in to branch-2. Can you please provide a patch of branch-2 instead of targeting 2.4?

        Show
        Haohui Mai added a comment - Downgrading to minor since documentation issues block no functionalities. By the way, since several major changes (e.g., HDFS-5978 and HDFS-6164 ) have been committed in to branch-2. Can you please provide a patch of branch-2 instead of targeting 2.4?
        Haohui Mai made changes -
        Target Version/s 2.4.0 [ 12326143 ] 2.5.0 [ 12326264 ]
        Haohui Mai made changes -
        Priority Blocker [ 1 ] Minor [ 4 ]
        Akira AJISAKA made changes -
        Summary Update OfflineImageViewer document Update OfflineImageViewer document for branch-2.4
        Priority Major [ 3 ] Blocker [ 1 ]
        Hide
        Akira AJISAKA added a comment -

        Haohui Mai, would you please review the patch for the 2.4 release?
        Also, we should also update the document for branch-2 later.

        Show
        Akira AJISAKA added a comment - Haohui Mai , would you please review the patch for the 2.4 release? Also, we should also update the document for branch-2 later.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12630161/HDFS-5865.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.fs.loadGenerator.TestLoadGenerator

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6197//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6197//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12630161/HDFS-5865.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.loadGenerator.TestLoadGenerator +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6197//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6197//console This message is automatically generated.
        Akira AJISAKA made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Affects Version/s 2.4.0 [ 12326143 ]
        Affects Version/s 3.0.0 [ 12320356 ]
        Target Version/s 3.0.0 [ 12320356 ] 2.4.0 [ 12326143 ]
        Akira AJISAKA made changes -
        Priority Minor [ 4 ] Major [ 3 ]
        Description The Offline Image Viewer document describes "Currently valid options are {{Ls}}, {{XML}}, and {{Indented}}" in {{--processor}} option, but now valid options are {{Ls}}, {{XML}}, and {{FileDistribution}}. OfflineImageViewer is renewed to handle the new format of fsimage by HDFS-5698 (fsimage in protobuf).

        We should document followings:

        * The tool can handle the layout version of Hadoop 2.4 and up. (If you want to handle the older version, you can use OfflineImageViewer of Hadoop 2.3)
        * Remove deprecated options such as Delimited and Indented processor.
        Akira AJISAKA made changes -
        Summary Document 'FileDistribution' argument in 'hdfs oiv --processor' option Update OfflineImageViewer document
        Akira AJISAKA made changes -
        Attachment HDFS-5865.patch [ 12630161 ]
        Hide
        Akira AJISAKA added a comment -

        Attaching a patch.

        Show
        Akira AJISAKA added a comment - Attaching a patch.
        Akira AJISAKA made changes -
        Assignee Akira AJISAKA [ ajisakaa ]
        Akira AJISAKA made changes -
        Field Original Value New Value
        Summary Document some arguments in 'hdfs oiv --processor' option Document 'FileDistribution' argument in 'hdfs oiv --processor' option
        Affects Version/s 3.0.0 [ 12320356 ]
        Affects Version/s 2.2.0 [ 12325049 ]
        Target Version/s 2.4.0 [ 12326143 ] 3.0.0 [ 12320356 ]
        Priority Major [ 3 ] Minor [ 4 ]
        Description The Offline Image Viewer document now describes "Currently valid options are {{Ls}}, {{XML}}, and {{Indented}}" in {{--processor}} option, but there're more options such as {{Delimited}}, {{FileDistribution}}, and {{NameDistribution}}. The Offline Image Viewer document describes "Currently valid options are {{Ls}}, {{XML}}, and {{Indented}}" in {{--processor}} option, but now valid options are {{Ls}}, {{XML}}, and {{FileDistribution}}.
        Akira AJISAKA created issue -

          People

          • Assignee:
            Akira AJISAKA
            Reporter:
            Akira AJISAKA
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development