Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5698

Use protobuf to serialize / deserialize FSImage

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4.0
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Use protobuf to serialize/deserialize the FSImage.

      Description

      Currently, the code serializes FSImage using in-house serialization mechanisms. There are a couple disadvantages of the current approach:

      1. Mixing the responsibility of reconstruction and serialization / deserialization. The current code paths of serialization / deserialization have spent a lot of effort on maintaining compatibility. What is worse is that they are mixed with the complex logic of reconstructing the namespace, making the code difficult to follow.
      2. Poor documentation of the current FSImage format. The format of the FSImage is practically defined by the implementation. An bug in implementation means a bug in the specification. Furthermore, it also makes writing third-party tools quite difficult.
      3. Changing schemas is non-trivial. Adding a field in FSImage requires bumping the layout version every time. Bumping out layout version requires (1) the users to explicitly upgrade the clusters, and (2) putting new code to maintain backward compatibility.

      This jira proposes to use protobuf to serialize the FSImage. Protobuf has been used to serialize / deserialize the RPC message in Hadoop.

      Protobuf addresses all the above problems. It clearly separates the responsibility of serialization and reconstructing the namespace. The protobuf files document the current format of the FSImage. The developers now can add optional fields with ease, since the old code can always read the new FSImage.

      1. HDFS-5698.000.patch
        107 kB
        Haohui Mai
      2. HDFS-5698.001.patch
        175 kB
        Haohui Mai
      3. HDFS-5698.002.patch
        179 kB
        Haohui Mai
      4. HDFS-5698-design.pdf
        118 kB
        Haohui Mai
      5. HDFS-5698.003.patch
        178 kB
        Haohui Mai
      6. HDFS-5698.004.patch
        196 kB
        Haohui Mai
      7. HDFS-5698.005.patch
        178 kB
        Haohui Mai
      8. HDFS-5698.006.patch
        179 kB
        Haohui Mai
      9. HDFS-5698.007.patch
        179 kB
        Haohui Mai
      10. HDFS-5698-branch2.000.patch
        182 kB
        Haohui Mai

        Issue Links

        1.
        Save FSImage header in protobuf Sub-task Resolved Haohui Mai
         
        2.
        Implement compression in the HTTP server of SNN / SBN instead of FSImage Sub-task Resolved Unassigned
         
        3.
        Remove compression support from FSImage Sub-task Resolved Haohui Mai
         
        4.
        Serialize INode information in protobuf Sub-task Resolved Haohui Mai
         
        5.
        Use protobuf to serialize snapshot information Sub-task Resolved Jing Zhao
         
        6.
        Serialize information for token managers in protobuf Sub-task Resolved Haohui Mai
         
        7.
        Track progress when loading fsimage Sub-task Resolved Haohui Mai
         
        8.
        Serialize under-construction file information in FSImage Sub-task Resolved Jing Zhao
         
        9.
        Serialize CachePool directives in protobuf Sub-task Resolved Haohui Mai
         
        10.
        Compute the digest before loading FSImage Sub-task Resolved Haohui Mai
         
        11.
        Serialize symlink in protobuf Sub-task Resolved Haohui Mai
         
        12.
        Optimize the serialization of PermissionStatus Sub-task Resolved Haohui Mai
         
        13.
        Implement offline image viewer. Sub-task Resolved Haohui Mai
         
        14.
        Implement cancellation when saving FSImage Sub-task Resolved Haohui Mai
         
        15.
        Add a Type field in Snapshot DiffEntry's protobuf definition Sub-task Resolved Jing Zhao
         
        16.
        Update the stored edit logs to be consistent with the changes in HDFS-5698 branch Sub-task Resolved Haohui Mai
         
        17.
        Consolidate INodeReference into a separate section Sub-task Closed Jing Zhao
         
        18.
        Use PBHelper to serialize CacheDirectiveInfoExpirationProto Sub-task Resolved Haohui Mai
         
        19.
        LoadDelegator should use IOUtils.readFully() to read the magic header Sub-task Resolved Haohui Mai
         
        20.
        Add annotation for repeated fields in the protobuf definition Sub-task Resolved Haohui Mai
         
        21. Add suffix to generated protobuf class Sub-task Patch Available Tassapol Athiapinya
         
        22.
        Fixing findbugs and javadoc warnings in the HDFS-5698 branch Sub-task Resolved Haohui Mai
         
        23.
        The id of a CacheDirective instance does not get serialized in the protobuf-fsimage Sub-task Resolved Haohui Mai
         

          Activity

          Hide
          Zesheng Wu added a comment -
          Show
          Zesheng Wu added a comment - Thanks Suresh Srinivas .
          Hide
          Suresh Srinivas added a comment -

          Also upgrade with HA enabled was also not there in that.

          If you are upgrading to 2.4.0, upgrade with HA enabled is supported. As Vinayakumar B points out, it still requires cluster downtime. The current documentation for 2.4.0 covers how to upgrade the HA cluster.

          Show
          Suresh Srinivas added a comment - Also upgrade with HA enabled was also not there in that. If you are upgrading to 2.4.0, upgrade with HA enabled is supported. As Vinayakumar B points out, it still requires cluster downtime. The current documentation for 2.4.0 covers how to upgrade the HA cluster.
          Hide
          Zesheng Wu added a comment -

          Got it, thanks Vinayakumar B.

          Show
          Zesheng Wu added a comment - Got it, thanks Vinayakumar B .
          Hide
          Vinayakumar B added a comment -

          Hi zesheng, unfortunately upgrade suppprt without downtime was not implemented in 2.0.0. Also upgrade with HA enabled was also not there in that. Rolling upgrade support is now added and is available 2.4.1 onwards.

          Show
          Vinayakumar B added a comment - Hi zesheng, unfortunately upgrade suppprt without downtime was not implemented in 2.0.0. Also upgrade with HA enabled was also not there in that. Rolling upgrade support is now added and is available 2.4.1 onwards.
          Hide
          Zesheng Wu added a comment -

          Thanks Suresh Srinivas, I've spent some time on how to upgrade the hdfs cluster from 2.0.0 to 2.4.0, I found that we must stop the whole cluster at first and then upgrade it, and this will result in service downtime. I am wondering whether we can upgrade the cluster without service downtime or not, supposing that the target cluster is deployed with QJM based HA enabled.

          Show
          Zesheng Wu added a comment - Thanks Suresh Srinivas , I've spent some time on how to upgrade the hdfs cluster from 2.0.0 to 2.4.0, I found that we must stop the whole cluster at first and then upgrade it, and this will result in service downtime. I am wondering whether we can upgrade the cluster without service downtime or not, supposing that the target cluster is deployed with QJM based HA enabled.
          Hide
          Suresh Srinivas added a comment -

          Zesheng Wu, there is nothing special needed. The usual upgrade procedure will work. The namenode in 2.4.0 will be able to load the fsimage written in older releases. It will then start writing the new fsimage in the newer format.

          Show
          Suresh Srinivas added a comment - Zesheng Wu , there is nothing special needed. The usual upgrade procedure will work. The namenode in 2.4.0 will be able to load the fsimage written in older releases. It will then start writing the new fsimage in the newer format.
          Hide
          Zesheng Wu added a comment -

          Hi forks, as this jira changed the fsimage greatly compared to the 2.0.0 version, is there any guideline for how to upgrade a 2.0.0 cluster to 2.4.0?

          Show
          Zesheng Wu added a comment - Hi forks, as this jira changed the fsimage greatly compared to the 2.0.0 version, is there any guideline for how to upgrade a 2.0.0 cluster to 2.4.0?
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1699 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1699/)
          HDFS-5698. Update CHANGES.txt to move breakdown of tasks to correct release (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568582)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1699 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1699/ ) HDFS-5698 . Update CHANGES.txt to move breakdown of tasks to correct release (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568582 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1674 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1674/)
          HDFS-5698. Update CHANGES.txt to move breakdown of tasks to correct release (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568582)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1674 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1674/ ) HDFS-5698 . Update CHANGES.txt to move breakdown of tasks to correct release (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568582 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #482 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/482/)
          HDFS-5698. Update CHANGES.txt to move breakdown of tasks to correct release (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568582)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #482 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/482/ ) HDFS-5698 . Update CHANGES.txt to move breakdown of tasks to correct release (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568582 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5175 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5175/)
          HDFS-5698. Update CHANGES.txt to move breakdown of tasks to correct release (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568582)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5175 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5175/ ) HDFS-5698 . Update CHANGES.txt to move breakdown of tasks to correct release (arp: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1568582 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          Hide
          Arpit Agarwal added a comment -

          Merge was committed as svn r1568548.

          Show
          Arpit Agarwal added a comment - Merge was committed as svn r1568548.
          Hide
          Arpit Agarwal added a comment -

          I committed the merge patch to branch-2. Thanks Haohui!

          Show
          Arpit Agarwal added a comment - I committed the merge patch to branch-2. Thanks Haohui!
          Hide
          Haohui Mai added a comment -

          Merge patch for branch-2.

          Show
          Haohui Mai added a comment - Merge patch for branch-2.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1694 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1694/)
          HDFS-5698. Use protobuf to serialize / deserialize FSImage. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566359)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageCompression.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/LsrPBImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1694 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1694/ ) HDFS-5698 . Use protobuf to serialize / deserialize FSImage. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566359 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageCompression.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/LsrPBImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1669 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1669/)
          HDFS-5698. Use protobuf to serialize / deserialize FSImage. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566359)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageCompression.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/LsrPBImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1669 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1669/ ) HDFS-5698 . Use protobuf to serialize / deserialize FSImage. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566359 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageCompression.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/LsrPBImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #477 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/477/)
          HDFS-5698. Use protobuf to serialize / deserialize FSImage. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566359)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageCompression.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/LsrPBImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #477 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/477/ ) HDFS-5698 . Use protobuf to serialize / deserialize FSImage. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566359 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageCompression.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/LsrPBImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5133 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5133/)
          HDFS-5698. Use protobuf to serialize / deserialize FSImage. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566359)

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageCompression.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageUtil.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/LsrPBImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5133 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5133/ ) HDFS-5698 . Use protobuf to serialize / deserialize FSImage. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1566359 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenSecretManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/CacheManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageCompression.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatProtobuf.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageUtil.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeMap.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SaveNamespaceContext.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/DirectoryWithSnapshotFeature.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FSImageFormatPBSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/FileDistributionCalculator.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/ImageLoaderCurrent.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/LsrPBImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageViewerPB.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/fsimage.proto /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImageWithSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestRenameWithSnapshots.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshot.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/TestOfflineImageViewer.java /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml
          Hide
          Jing Zhao added a comment -

          I've merged this into trunk. Thanks for the contribution Haohui!

          Show
          Jing Zhao added a comment - I've merged this into trunk. Thanks for the contribution Haohui!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12627746/HDFS-5698.007.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6083//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6083//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12627746/HDFS-5698.007.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6083//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6083//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12627673/HDFS-5698.006.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.TestSafeMode
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
          org.apache.hadoop.hdfs.server.namenode.TestAuditLogs

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6071//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6071//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12627673/HDFS-5698.006.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.TestSafeMode org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives org.apache.hadoop.hdfs.server.namenode.TestAuditLogs +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6071//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6071//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12627570/HDFS-5698.005.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 3 warning messages.
          See https://builds.apache.org/job/PreCommit-HDFS-Build/6063//artifact/trunk/patchprocess/diffJavadocWarnings.txt for details.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6063//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/6063//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6063//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12627570/HDFS-5698.005.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. See https://builds.apache.org/job/PreCommit-HDFS-Build/6063//artifact/trunk/patchprocess/diffJavadocWarnings.txt for details. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6063//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/6063//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6063//console This message is automatically generated.
          Hide
          Haohui Mai added a comment -

          The v5 patch excludes changes that haven't been merged into the branch.

          Show
          Haohui Mai added a comment - The v5 patch excludes changes that haven't been merged into the branch.
          Hide
          Haohui Mai added a comment -

          Thanks very much for the detailed comments from Todd Lipcon.

          I've filed HDFS-5884, HDFS-5885 and HDFS-5887 to address the comments.

          Thanks very much for the suggestions of the performance improvement. I'll dig into it.

          My plan is to commit HDFS-5884 and HDFS-5885 before the merge, and to continue to improve the code in trunk. Does it make sense for you?

          would existing ImageVisitor implementation classes continue to work against the PB-ified image?

          The existing ImageVisitor implementation won't work with the PB FSImage.

          Show
          Haohui Mai added a comment - Thanks very much for the detailed comments from Todd Lipcon . I've filed HDFS-5884 , HDFS-5885 and HDFS-5887 to address the comments. Thanks very much for the suggestions of the performance improvement. I'll dig into it. My plan is to commit HDFS-5884 and HDFS-5885 before the merge, and to continue to improve the code in trunk. Does it make sense for you? would existing ImageVisitor implementation classes continue to work against the PB-ified image? The existing ImageVisitor implementation won't work with the PB FSImage.
          Hide
          Todd Lipcon added a comment -

          A few notes on the patch:

          +        is = new FileInputStream(file);
          +        if (is.read(magic) == magic.length
          

          Should use IOUtils.readFully here


          Can we rename INodeSection and the other nested proto classes to end in "PB" or "Proto"? It's helpful when reading the code to distinguish the generated protobuf classes from the other structures, and given that these inner classes get imported, it's not always obvious.


          Performance-wise, I think you can really improve things by re-using protobuf objects. In particular, rather than doing something like:

          +      INodeSection.INodeReference ref = INodeSection.INodeReference
          +          .parseDelimitedFrom(in);
          +      return loadINodeReference(ref, dir);
          

          you can make a thread-local INodeSection.INodeReference.Builder object (similar to how we use thread-local ops in the editlog loader code). Then use Builder.mergeDelimitedFrom instead of the static parseDelimitedFrom method. You can check isInitialized() after this to ensure that all of the required fields are present, and then use the builder itself to read the fields. This avoids repeated object allocation/deallocation costs without having to resort to manual parsing that you mention in the design doc.

          The generated code also has a handy "FooProtoOrBuilder" interface that both the generated PB and its builder implement, with all of the appropriate getters. The code that actually handles constructing HDFS objects from PBs could easily take this interface.


          For many of the repeated int64 fields, you should probably use the [packed=true] option in the protobuf definition. This saves a good amount of space and probably improves decoding performance as well.


          One question: would existing ImageVisitor implementation classes continue to work against the PB-ified image? My reading of the patch is that they wouldn't, but would be nice to confirm.


          I don't think any of the above needs to block the merge, but the format-breaking one (packed=true) should probably be done sooner rather than later.

          Show
          Todd Lipcon added a comment - A few notes on the patch: + is = new FileInputStream(file); + if (is.read(magic) == magic.length Should use IOUtils.readFully here Can we rename INodeSection and the other nested proto classes to end in "PB" or "Proto"? It's helpful when reading the code to distinguish the generated protobuf classes from the other structures, and given that these inner classes get imported, it's not always obvious. Performance-wise, I think you can really improve things by re-using protobuf objects. In particular, rather than doing something like: + INodeSection.INodeReference ref = INodeSection.INodeReference + .parseDelimitedFrom(in); + return loadINodeReference(ref, dir); you can make a thread-local INodeSection.INodeReference.Builder object (similar to how we use thread-local ops in the editlog loader code). Then use Builder.mergeDelimitedFrom instead of the static parseDelimitedFrom method. You can check isInitialized() after this to ensure that all of the required fields are present, and then use the builder itself to read the fields. This avoids repeated object allocation/deallocation costs without having to resort to manual parsing that you mention in the design doc. The generated code also has a handy "FooProtoOrBuilder" interface that both the generated PB and its builder implement, with all of the appropriate getters. The code that actually handles constructing HDFS objects from PBs could easily take this interface. For many of the repeated int64 fields, you should probably use the [packed=true] option in the protobuf definition. This saves a good amount of space and probably improves decoding performance as well. One question: would existing ImageVisitor implementation classes continue to work against the PB-ified image? My reading of the patch is that they wouldn't, but would be nice to confirm. I don't think any of the above needs to block the merge, but the format-breaking one (packed=true) should probably be done sooner rather than later.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12626185/HDFS-5698.003.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5995//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5995//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5995//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12626185/HDFS-5698.003.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5995//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5995//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5995//console This message is automatically generated.
          Hide
          Haohui Mai added a comment -

          Remove CHANGE-5698.txt from the patch to avoid release audit warnings.

          Show
          Haohui Mai added a comment - Remove CHANGE-5698.txt from the patch to avoid release audit warnings.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12626179/HDFS-5698-design.pdf
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5994//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12626179/HDFS-5698-design.pdf against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5994//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12626179/HDFS-5698-design.pdf
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5993//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12626179/HDFS-5698-design.pdf against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5993//console This message is automatically generated.
          Hide
          Haohui Mai added a comment -

          The pdf describes the design and the experience of the new, protobuf-based FSImage format.

          Show
          Haohui Mai added a comment - The pdf describes the design and the experience of the new, protobuf-based FSImage format.
          Hide
          Haohui Mai added a comment -

          Rebased consolidated patch.

          Show
          Haohui Mai added a comment - Rebased consolidated patch.
          Hide
          Haohui Mai added a comment -

          I prototyped a parallelized version of the FSImageLoader. Here is the number running on the same environment:

          Size in Old 512M 1G 2G 4G 8G
          Loading in Old (ms) 12819 24664 48240 114090 307689
          Loading in PB(Parallel) (ms) 17927 32997 64581 138306 373391

          The main changes of the prototype are:

          1. Compute the MD5 checksum in a separate thread.
          2. Parallelize the construction of INodes (with 6 threads)
          3. Use a dedicated thread to update the blocks map.

          Currently I haven't parallelize the construction of INodeDirectory yet, but it seems to me that the performance numbers are reasonable (recall that coming out the safe mode automatically will take 30 seconds). I think we can fine tune the performance after the work is merged into trunk.

          Show
          Haohui Mai added a comment - I prototyped a parallelized version of the FSImageLoader. Here is the number running on the same environment: Size in Old 512M 1G 2G 4G 8G Loading in Old (ms) 12819 24664 48240 114090 307689 Loading in PB(Parallel) (ms) 17927 32997 64581 138306 373391 The main changes of the prototype are: Compute the MD5 checksum in a separate thread. Parallelize the construction of INodes (with 6 threads) Use a dedicated thread to update the blocks map. Currently I haven't parallelize the construction of INodeDirectory yet, but it seems to me that the performance numbers are reasonable (recall that coming out the safe mode automatically will take 30 seconds). I think we can fine tune the performance after the work is merged into trunk.
          Hide
          Suresh Srinivas added a comment -

          It'll be important to know if NN's with huge images will be unable to load their images w/o more heap allocation.

          All the objects created are short lived. Hence this should not affect NN heap allocation. However, it would be interesting to see the time spent in GC and rate of garbage creation.

          Show
          Suresh Srinivas added a comment - It'll be important to know if NN's with huge images will be unable to load their images w/o more heap allocation. All the objects created are short lived. Hence this should not affect NN heap allocation. However, it would be interesting to see the time spent in GC and rate of garbage creation.
          Hide
          Daryn Sharp added a comment -

          You may want to investigate if the inodemap will perform better with a ConcurrentHashMap than a LightWeightGSet. That will increase the parallelism of the map insertion. I think the gset was chosen for memory concerns.

          Assuming you plan to parallelize the parent/child linkages, I think the addChild may need to be in a synchronized block unless the inodeMap is made concurrent. I'm not a snapshot expert, but I wonder how thread-safe the snapshot manager is. Are the directory diffs constructed "on the fly" during addition of the children, or are they stored separately in the fsimage?

          We just need to be certain it's actually feasible to offset a ~2X increase in load time.

          Also, did you happen to gather heap usage statistics? Is part of the load increase maybe due to increased GC? It'll be important to know if NN's with huge images will be unable to load their images w/o more heap allocation.

          Show
          Daryn Sharp added a comment - You may want to investigate if the inodemap will perform better with a ConcurrentHashMap than a LightWeightGSet . That will increase the parallelism of the map insertion. I think the gset was chosen for memory concerns. Assuming you plan to parallelize the parent/child linkages, I think the addChild may need to be in a synchronized block unless the inodeMap is made concurrent. I'm not a snapshot expert, but I wonder how thread-safe the snapshot manager is. Are the directory diffs constructed "on the fly" during addition of the children, or are they stored separately in the fsimage? We just need to be certain it's actually feasible to offset a ~2X increase in load time. Also, did you happen to gather heap usage statistics? Is part of the load increase maybe due to increased GC? It'll be important to know if NN's with huge images will be unable to load their images w/o more heap allocation.
          Hide
          Haohui Mai added a comment -

          Our profiling results show that the parsing the bytes and constructing the protobuf objects take significant amount of time.

          The work is parallelized like the following:

          while (has data) {
            bytes[] data = read();
            thread_pool.submit(parse_data(data));
          }
          
          parse_data_for_inode(data) {
            INode inode = construct(data);
            synchronized(inodemap) {
              inodemap.add(inode);
            }
            block_map_thread_pool.submit(update_block_map(data));
          }
          
          parse_data_for_inode_dir(data) {
            foreach (children : data.getChildren())
              inodemap.get(data.getparent()).addChild(inodemap.get(children))
          }
          

          Two things are worth noting. (1) The contention only happens when adding the inode into the inodemap. (2) Updating the block maps happens in parallel. Our profiling results show that updating the block maps can take up to 20% of the execution time. The latency can be hidden in the above implementation.

          I've only tested an early prototype on my laptop. With 4 threads it brings the load latency comparable to the old format. To report comparable numbers, however, I'll need to update the code and rerun the test on the machine that I ran my previous tests.

          Show
          Haohui Mai added a comment - Our profiling results show that the parsing the bytes and constructing the protobuf objects take significant amount of time. The work is parallelized like the following: while (has data) { bytes[] data = read(); thread_pool.submit(parse_data(data)); } parse_data_for_inode(data) { INode inode = construct(data); synchronized (inodemap) { inodemap.add(inode); } block_map_thread_pool.submit(update_block_map(data)); } parse_data_for_inode_dir(data) { foreach (children : data.getChildren()) inodemap.get(data.getparent()).addChild(inodemap.get(children)) } Two things are worth noting. (1) The contention only happens when adding the inode into the inodemap. (2) Updating the block maps happens in parallel. Our profiling results show that updating the block maps can take up to 20% of the execution time. The latency can be hidden in the above implementation. I've only tested an early prototype on my laptop. With 4 threads it brings the load latency comparable to the old format. To report comparable numbers, however, I'll need to update the code and rerun the test on the machine that I ran my previous tests.
          Hide
          Kihwal Lee added a comment -

          Thanks for running tests and sharing the numbers. I did some testing In the past and the loading speed was about 30MB/sec at best. I/O wasn't the bottleneck. THP and CompressedOOPS help a bit, but in the end the bottleneck was java object creations. Due to the way things are serialized, multi-threaded loading wasn't feasible.

          Now that we have the inode section and the inode directory section separated, parallelism can be added for loading each section. Please share your implementation ideas. The parallelism may come out far less than expected due to internal locks. So it will be great if a rough prototype & testing is done to show what's attainable. Do you already have numbers for how long it took to load each section?

          Show
          Kihwal Lee added a comment - Thanks for running tests and sharing the numbers. I did some testing In the past and the loading speed was about 30MB/sec at best. I/O wasn't the bottleneck. THP and CompressedOOPS help a bit, but in the end the bottleneck was java object creations. Due to the way things are serialized, multi-threaded loading wasn't feasible. Now that we have the inode section and the inode directory section separated, parallelism can be added for loading each section. Please share your implementation ideas. The parallelism may come out far less than expected due to internal locks. So it will be great if a rough prototype & testing is done to show what's attainable. Do you already have numbers for how long it took to load each section?
          Hide
          Haohui Mai added a comment -

          I took a fsimage from a production cluster, and scaled it down to different sizes to evaluate the performance and the size impact.

          I ran the test on a machine that has an 8-core Xeon E5530 CPU @ 2.4GHz, 24G memory, 2TB SATA 3 drive @ 7200 rpm. The machine is running RHEL 6.2, Java 1.6. The JVM has a maximum heap size of 20G, and it runs the concurrent mark and sweep GC.

          Here are the numbers:

          Size in Old 512M 1G 2G 4G 8G
          Size in PB 469M 950M 1.9G 3.7G 7.0G
          Saving in Old (ms) 14678 28991 60520 96894 160878
          Saving in PB (ms) 14709 16746 32623 83645 168617
          Loading in Old (ms) 12819 24664 48240 114090 307689
          Loading in PB (ms) 28268 43205 87060 266681 491605

          The first two rows show the size of the fsimage in both the old and the new format respectively. The third and the forth row show the time of saving the fsimage in two different formats, and the last two rows show the time of loading the fsimage in two different format.

          The new fsimage format is slightly more compact. The code writes the new fsimage slightly faster. Currently the new fsimage format loads slower. However, in the new format most of the loading process can be parallelized. I plan to introduce this feature after the branch is merged.

          Show
          Haohui Mai added a comment - I took a fsimage from a production cluster, and scaled it down to different sizes to evaluate the performance and the size impact. I ran the test on a machine that has an 8-core Xeon E5530 CPU @ 2.4GHz, 24G memory, 2TB SATA 3 drive @ 7200 rpm. The machine is running RHEL 6.2, Java 1.6. The JVM has a maximum heap size of 20G, and it runs the concurrent mark and sweep GC. Here are the numbers: Size in Old 512M 1G 2G 4G 8G Size in PB 469M 950M 1.9G 3.7G 7.0G Saving in Old (ms) 14678 28991 60520 96894 160878 Saving in PB (ms) 14709 16746 32623 83645 168617 Loading in Old (ms) 12819 24664 48240 114090 307689 Loading in PB (ms) 28268 43205 87060 266681 491605 The first two rows show the size of the fsimage in both the old and the new format respectively. The third and the forth row show the time of saving the fsimage in two different formats, and the last two rows show the time of loading the fsimage in two different format. The new fsimage format is slightly more compact. The code writes the new fsimage slightly faster. Currently the new fsimage format loads slower. However, in the new format most of the loading process can be parallelized. I plan to introduce this feature after the branch is merged.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12625413/HDFS-5698.001.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5952//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5952//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5952//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12625413/HDFS-5698.001.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 7 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5952//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5952//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5952//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          I like the premise of this jira but like Kihwal, I too am interested in the performance/size impacts. My concern is edit logging is already imposing a non-trivial penalty to throughput. The longer we spend constructing an edit log entry, the longer the namespace is write locked. The larger the edits, the fewer edit log entries can be batched synced which both incurs more i/o penalties and ties up the rpc handler threads longer. It would be nice if you would gather some statistics.

          Show
          Daryn Sharp added a comment - I like the premise of this jira but like Kihwal, I too am interested in the performance/size impacts. My concern is edit logging is already imposing a non-trivial penalty to throughput. The longer we spend constructing an edit log entry, the longer the namespace is write locked. The larger the edits, the fewer edit log entries can be batched synced which both incurs more i/o penalties and ties up the rpc handler threads longer. It would be nice if you would gather some statistics.
          Hide
          Kihwal Lee added a comment -

          What will be the performance/size implications of this?

          Show
          Kihwal Lee added a comment - What will be the performance/size implications of this?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12624060/HDFS-5698.000.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 javadoc. The javadoc tool appears to have generated 3 warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 48 new Findbugs (version 1.3.9) warnings.

          -1 release audit. The applied patch generated 1 release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.hdfs.TestCrcCorruption
          org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace
          org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer
          org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
          org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
          org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5923//testReport/
          Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5923//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5923//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5923//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12624060/HDFS-5698.000.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. -1 javadoc . The javadoc tool appears to have generated 3 warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 48 new Findbugs (version 1.3.9) warnings. -1 release audit . The applied patch generated 1 release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.hdfs.TestCrcCorruption org.apache.hadoop.hdfs.server.namenode.TestSaveNamespace org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/5923//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5923//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/5923//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5923//console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          I added HDFS-5698 as a version for HDFS project.

          Show
          Suresh Srinivas added a comment - I added HDFS-5698 as a version for HDFS project.
          Hide
          Arpit Agarwal added a comment -

          Created a new branch HDFS-5698 for this work (svn commit: r1554967). Can someone with Jira admin privileges please create a Jira version?

          Show
          Arpit Agarwal added a comment - Created a new branch HDFS-5698 for this work (svn commit: r1554967). Can someone with Jira admin privileges please create a Jira version?
          Hide
          Colin Patrick McCabe added a comment -

          This is a good idea. I agree with Todd that we'll still need feature flags, to avoid data loss when starting up with a different version. Also, we should keep the current deserialization logic around so that we have backwards compatibility, the same as we have done in the past. That way, this can be quickly and compatibly backported to branch-2, rather than maintaining a fork of the whole FSImage deserialization code in trunk.

          Show
          Colin Patrick McCabe added a comment - This is a good idea. I agree with Todd that we'll still need feature flags, to avoid data loss when starting up with a different version. Also, we should keep the current deserialization logic around so that we have backwards compatibility, the same as we have done in the past. That way, this can be quickly and compatibly backported to branch-2, rather than maintaining a fork of the whole FSImage deserialization code in trunk.
          Hide
          Todd Lipcon added a comment -

          With the proposal we should make sure we still have a way of preventing loss of data using an older version to read a newer version file. That is to say, it's important to still have a version number, or the "feature flags" (HDFS-5223) implemented. Otherwise, you may be relying on some new feature present in an optional field, and restarting with an old NN would silently ignore and delete that data on next checkpoint.

          Show
          Todd Lipcon added a comment - With the proposal we should make sure we still have a way of preventing loss of data using an older version to read a newer version file. That is to say, it's important to still have a version number, or the "feature flags" ( HDFS-5223 ) implemented. Otherwise, you may be relying on some new feature present in an optional field, and restarting with an old NN would silently ignore and delete that data on next checkpoint.
          Hide
          Andrew Wang added a comment -

          +1, I'd really like to see this in 3.0. It'd be great if we could protobuf the edits as well, since that code is flawed for the same reasons as above.

          Show
          Andrew Wang added a comment - +1, I'd really like to see this in 3.0. It'd be great if we could protobuf the edits as well, since that code is flawed for the same reasons as above.
          Hide
          Zesheng Wu added a comment -

          This sounds great~

          Show
          Zesheng Wu added a comment - This sounds great~
          Hide
          Andrew Purtell added a comment -

          +1, this would be great

          Show
          Andrew Purtell added a comment - +1, this would be great
          Hide
          Vinayakumar B added a comment -

          Thanks for filing this Jira Haohui Mai. Looks like its going to make maintainance of fsimage very easy.

          Show
          Vinayakumar B added a comment - Thanks for filing this Jira Haohui Mai . Looks like its going to make maintainance of fsimage very easy.
          Hide
          Fengdong Yu added a comment -

          Hi,
          So we don't need to upgrade cluster after this was fixed, right? and we also can do rolling upgrade with this feature support, also right?

          Show
          Fengdong Yu added a comment - Hi, So we don't need to upgrade cluster after this was fixed, right? and we also can do rolling upgrade with this feature support, also right?

            People

            • Assignee:
              Haohui Mai
              Reporter:
              Haohui Mai
            • Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development