Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2212

Refactor double-buffering code out of EditLogOutputStreams

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This is a small cleanup that makes EditLogFileOutputStream and EditLogBackupOutputStream more consistent with each other on how they buffer edits. It simply refactors the double-buffering behavior into a new class.

      1. hdfs-2212.txt
        20 kB
        Todd Lipcon

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-1073-branch #23 (See https://builds.apache.org/job/Hadoop-Hdfs-1073-branch/23/)
        Merge trunk into HDFS-1073.

        Resolved several conflicts due to merge of HDFS-2149 and HDFS-2212.
        Changes during resolution were:

        • move the writing of the transaction ID out of EditLogOutputStream to
          FSEditLogOp.Writer to match trunk's organization
        • remove JSPOOL related FsEditLogOp subclasses, add LogSegmentOp subclasses
        • modify TestEditLogJournalFailures to not keep trying to use streams after
          the simulated halt, since newer stricter assertions caused these writes to
          fail

        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152128
        Files :

        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileAppend2.java
        • /hadoop/common/branches/HDFS-1073/hdfs/CHANGES.txt
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileStatus.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/docs/src/documentation/content/xdocs/hdfsproxy.xml
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestComputeInvalidateWork.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/contrib/build.xml
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDatanodeDeath.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreationDelete.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileAppend3.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/DFSClient.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileConcurrentReader.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/docs/src/documentation/content/xdocs/site.xml
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/unit/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithMultipleNameNodes.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDecommission.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestEditsDoubleBuffer.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestLeaseRecovery2.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/common/Storage.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/webapps/datanode
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileAppend4.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/UpgradeUtilities.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/contrib/hdfsproxy
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreation.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/aop/org/apache/hadoop/hdfs/TestFiPipelines.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCorruption.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestComputeInvalidateWork.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestPipelines.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/webapps/hdfs
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestReadWhileWriting.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/webapps/secondary
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestRenameWhileOpen.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreationEmpty.java
        • /hadoop/common/branches/HDFS-1073/hdfs
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/c++/libhdfs
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestStorageRestore.java
        • /hadoop/common/branches/HDFS-1073/hdfs/build.xml
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreationClient.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditsDoubleBuffer.java
        • /hadoop/common/branches/HDFS-1073/hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-1073-branch #23 (See https://builds.apache.org/job/Hadoop-Hdfs-1073-branch/23/ ) Merge trunk into HDFS-1073 . Resolved several conflicts due to merge of HDFS-2149 and HDFS-2212 . Changes during resolution were: move the writing of the transaction ID out of EditLogOutputStream to FSEditLogOp.Writer to match trunk's organization remove JSPOOL related FsEditLogOp subclasses, add LogSegmentOp subclasses modify TestEditLogJournalFailures to not keep trying to use streams after the simulated halt, since newer stricter assertions caused these writes to fail todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1152128 Files : /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileAppend2.java /hadoop/common/branches/ HDFS-1073 /hdfs/CHANGES.txt /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileStatus.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/docs/src/documentation/content/xdocs/hdfsproxy.xml /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestComputeInvalidateWork.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/contrib/build.xml /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDatanodeDeath.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreationDelete.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSUpgradeFromImage.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestClusterId.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileAppend3.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/common/JspHelper.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/DFSClient.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestEditLogJournalFailures.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileConcurrentReader.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/docs/src/documentation/content/xdocs/site.xml /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/unit/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/balancer/TestBalancerWithMultipleNameNodes.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestDecommission.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestEditsDoubleBuffer.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestLeaseRecovery2.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/common/Storage.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/webapps/datanode /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestMultiThreadedHflush.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileAppend4.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/UpgradeUtilities.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/contrib/hdfsproxy /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/BlockPoolSliceStorage.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreation.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/aop/org/apache/hadoop/hdfs/TestFiPipelines.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCorruption.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestComputeInvalidateWork.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestPipelines.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/webapps/hdfs /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestReadWhileWriting.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/webapps/secondary /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestRenameWhileOpen.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreationEmpty.java /hadoop/common/branches/ HDFS-1073 /hdfs /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestBlockReport.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/c++/libhdfs /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestStorageRestore.java /hadoop/common/branches/ HDFS-1073 /hdfs/build.xml /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestFileCreationClient.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditsDoubleBuffer.java /hadoop/common/branches/ HDFS-1073 /hdfs/src/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #812 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/812/)
        HDFS-2212. Refactor double-buffering code out of EditLogOutputStreams. Contributed by Todd Lipcon

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151736
        Files :

        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestEditsDoubleBuffer.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditsDoubleBuffer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #812 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/812/ ) HDFS-2212 . Refactor double-buffering code out of EditLogOutputStreams. Contributed by Todd Lipcon eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151736 Files : /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestEditsDoubleBuffer.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditsDoubleBuffer.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #738 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/738/)
        HDFS-2212. Refactor double-buffering code out of EditLogOutputStreams. Contributed by Todd Lipcon

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151736
        Files :

        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java
        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestEditsDoubleBuffer.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditsDoubleBuffer.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #738 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/738/ ) HDFS-2212 . Refactor double-buffering code out of EditLogOutputStreams. Contributed by Todd Lipcon eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1151736 Files : /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileOutputStream.java /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestEditsDoubleBuffer.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupOutputStream.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/EditsDoubleBuffer.java
        Hide
        Todd Lipcon added a comment -

        Thanks, Eli. btw, I ran the related unit tests locally and they passed (Hudson boxes are down - see my email to common-dev)

        Show
        Todd Lipcon added a comment - Thanks, Eli. btw, I ran the related unit tests locally and they passed (Hudson boxes are down - see my email to common-dev)
        Hide
        Eli Collins added a comment -

        I've committed this. Thanks Todd.

        Show
        Eli Collins added a comment - I've committed this. Thanks Todd.
        Hide
        Eli Collins added a comment -

        Ah, missed that. +1 lgtm

        Show
        Eli Collins added a comment - Ah, missed that. +1 lgtm
        Hide
        Todd Lipcon added a comment -

        It's not removed, it just got moved to FSEditLogOp.Writer so it happens in the same place as the opcode (and the txid in 1073 branch)

        Show
        Todd Lipcon added a comment - It's not removed, it just got moved to FSEditLogOp.Writer so it happens in the same place as the opcode (and the txid in 1073 branch)
        Hide
        Eli Collins added a comment -

        Why is it OK to remove the checksumming from ELFOS#write?

        Show
        Eli Collins added a comment - Why is it OK to remove the checksumming from ELFOS#write?
        Hide
        Todd Lipcon added a comment -

        btw, I should note that this doesn't work with the BackupNode, but the BN is broken in trunk as is – since it doesn't handle checksummed edits correctly. This is addressed in 1073 which is merging any day now.

        Show
        Todd Lipcon added a comment - btw, I should note that this doesn't work with the BackupNode, but the BN is broken in trunk as is – since it doesn't handle checksummed edits correctly. This is addressed in 1073 which is merging any day now.
        Hide
        Todd Lipcon added a comment -

        Fairly simple patch. I threw in a true unit test while I was at it (why not?)

        Show
        Todd Lipcon added a comment - Fairly simple patch. I threw in a true unit test while I was at it (why not?)

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Todd Lipcon
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development