Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.0
    • Fix Version/s: 2.5.0
    • Component/s: documentation
    • Labels:
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      OfflineImageViewer is renewed to handle the new format of fsimage by HDFS-5698 (fsimage in protobuf).

      We should document the followings:

      • The tool can handle the layout version of Hadoop 2.4 and up. (If you want to handle the older version, you can use OfflineImageViewer of Hadoop 2.3)
      • Delimited, Indented, and Ls processor were removed.
      • A new Web processor, which supersedes the Ls processor, was added.
      1. HDFS-5865.2.patch
        23 kB
        Akira AJISAKA
      2. HDFS-5865.patch
        18 kB
        Akira AJISAKA

        Activity

        Hide
        Akira AJISAKA added a comment -

        Attaching a patch.

        Show
        Akira AJISAKA added a comment - Attaching a patch.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12630161/HDFS-5865.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

        org.apache.hadoop.fs.loadGenerator.TestLoadGenerator

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6197//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6197//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12630161/HDFS-5865.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.fs.loadGenerator.TestLoadGenerator +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6197//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6197//console This message is automatically generated.
        Hide
        Akira AJISAKA added a comment -

        Haohui Mai, would you please review the patch for the 2.4 release?
        Also, we should also update the document for branch-2 later.

        Show
        Akira AJISAKA added a comment - Haohui Mai , would you please review the patch for the 2.4 release? Also, we should also update the document for branch-2 later.
        Hide
        Haohui Mai added a comment -

        Downgrading to minor since documentation issues block no functionalities.

        By the way, since several major changes (e.g., HDFS-5978 and HDFS-6164) have been committed in to branch-2. Can you please provide a patch of branch-2 instead of targeting 2.4?

        Show
        Haohui Mai added a comment - Downgrading to minor since documentation issues block no functionalities. By the way, since several major changes (e.g., HDFS-5978 and HDFS-6164 ) have been committed in to branch-2. Can you please provide a patch of branch-2 instead of targeting 2.4?
        Hide
        Akira AJISAKA added a comment -

        I can provide a patch of branch-2, but I think 2.4 is the first. We should document the deprecated options and backward incompatibility because major changes have been committed in 2.4 branch also.

        Show
        Akira AJISAKA added a comment - I can provide a patch of branch-2, but I think 2.4 is the first. We should document the deprecated options and backward incompatibility because major changes have been committed in 2.4 branch also.
        Hide
        Akira AJISAKA added a comment -

        Since 2.4.0-rc0 will come soon, I'll update the description and create a patch for branch-2.

        Show
        Akira AJISAKA added a comment - Since 2.4.0-rc0 will come soon, I'll update the description and create a patch for branch-2.
        Hide
        Haohui Mai added a comment - - edited
        +| <<<-i>>>\u007c<<<--inputFile>>> <input file> | Specify the input fsimage file
        +| <<<-o>>>\u007c<<<--outputFile>>> <output file> | Specify the output filename,
        +| <<<-p>>>\u007c<<<--processor>>> <processor> | Specify the image processor to
        +| <<<-h>>>\u007c<<<--help>>>| Display the tool usage and help information and
        ...
        

        why using \u007c instead of |, which is in the old document?

        There are several paragraphs that are specific to the ls output processor haven't been cleaned up yet. For example:

           This will create a file named fsimage.txt in the current directory
           using the Ls output processor. For very large image files, this process
           may take several minutes.
        
           One can specify which output processor via the command-line switch -p.
           For instance:
        
        Show
        Haohui Mai added a comment - - edited +| <<<-i>>>\u007c<<<--inputFile>>> <input file> | Specify the input fsimage file +| <<<-o>>>\u007c<<<--outputFile>>> <output file> | Specify the output filename, +| <<<-p>>>\u007c<<<--processor>>> <processor> | Specify the image processor to +| <<<-h>>>\u007c<<<--help>>>| Display the tool usage and help information and ... why using \u007c instead of | , which is in the old document? There are several paragraphs that are specific to the ls output processor haven't been cleaned up yet. For example: This will create a file named fsimage.txt in the current directory using the Ls output processor. For very large image files, this process may take several minutes. One can specify which output processor via the command-line switch -p. For instance:
        Hide
        Akira AJISAKA added a comment -

        Thanks for the review, Haohui.

        why using \u007c instead of |, which is in the old document?

        I thought | was not available in a cell, but I confirmed it was available. I'll use | in the next patch.

        There are several paragraphs that are specific to the ls output processor haven't been cleaned up yet.

        Ls was not cleaned up yet in the patch because this patch was for branch-2.4. I'll update the patch.

        Show
        Akira AJISAKA added a comment - Thanks for the review, Haohui. why using \u007c instead of |, which is in the old document? I thought | was not available in a cell, but I confirmed it was available. I'll use | in the next patch. There are several paragraphs that are specific to the ls output processor haven't been cleaned up yet. Ls was not cleaned up yet in the patch because this patch was for branch-2.4. I'll update the patch.
        Hide
        Akira AJISAKA added a comment -

        Attaching a patch to add the description of the Web processor.

        Show
        Akira AJISAKA added a comment - Attaching a patch to add the description of the Web processor.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12641879/HDFS-5865.2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6731//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6731//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641879/HDFS-5865.2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +0 tests included . The patch appears to be a documentation patch that doesn't require tests. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6731//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6731//console This message is automatically generated.
        Hide
        Haohui Mai added a comment -

        +1

        Show
        Haohui Mai added a comment - +1
        Hide
        Haohui Mai added a comment -

        I've committed the patch to trunk and branch-2. Thanks Akira AJISAKA for the contribution.

        Show
        Haohui Mai added a comment - I've committed the patch to trunk and branch-2. Thanks Akira AJISAKA for the contribution.
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #5574 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5574/)
        HDFS-5865. Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5574 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5574/ ) HDFS-5865 . Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #552 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/552/)
        HDFS-5865. Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #552 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/552/ ) HDFS-5865 . Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Hdfs-trunk #1743 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1743/)
        HDFS-5865. Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1743 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1743/ ) HDFS-5865 . Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1769 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1769/)
        HDFS-5865. Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1769 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1769/ ) HDFS-5865 . Update OfflineImageViewer document. Contributed by Akira Ajisaka. (wheat9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1590100 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsImageViewer.apt.vm

          People

          • Assignee:
            Akira AJISAKA
            Reporter:
            Akira AJISAKA
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development