Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2106

Umbrella JIRA for separating block management and name space management in NameNode

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None

      Description

      Namenode has two main components, block management and namespace management. The ultimate goal is to allow BM and NM running in separated servers. The short-term goal is to separate the codes.

      1. bm_api20110628.pdf
        54 kB
        Tsz Wo Nicholas Sze
      2. bm_api20110627.pdf
        53 kB
        Tsz Wo Nicholas Sze
      1.
      Move block management code to a package Sub-task Closed Tsz Wo Nicholas Sze
       
      2.
      Move datanode heartbeat handling to BlockManager Sub-task Closed Tsz Wo Nicholas Sze
       
      3.
      Move ReplicationMonitor to block management Sub-task Closed Uma Maheswara Rao G
       
      4.
      Move DecommissionManager to block management Sub-task Closed Tsz Wo Nicholas Sze
       
      5.
      Move Host2NodesMap to block management Sub-task Closed Tsz Wo Nicholas Sze
       
      6.
      Move cluster network topology to block management Sub-task Closed Tsz Wo Nicholas Sze
       
      7.
      Move dnsToSwitchMapping and hostsReader from FSNamesystem to DatanodeManager Sub-task Closed Tsz Wo Nicholas Sze
       
      8.
      Move datanodeMap from FSNamesystem to DatanodeManager Sub-task Closed Tsz Wo Nicholas Sze
       
      9.
      Move blockTokenSecretManager from FSNamesystem to BlockManager Sub-task Closed Uma Maheswara Rao G
       
      10.
      Set FSNamesystem.LOG to package private Sub-task Closed Tsz Wo Nicholas Sze
       
      11. Add read-write lock to DatanodeManager Sub-task Open Tsz Wo Nicholas Sze
       
      12.
      Move block and datanode code from FSNamesystem to BlockManager and DatanodeManager Sub-task Closed Tsz Wo Nicholas Sze
       
      13.
      Change UnderReplicatedBlocks from public to package private Sub-task Closed Tsz Wo Nicholas Sze
       
      14.
      Reduce access levels of the fields and methods in FSNamesystem Sub-task Closed Tsz Wo Nicholas Sze
       
      15.
      Remove unnecessary BlockTokenSecretManager fields/methods from BlockManager Sub-task Closed Tsz Wo Nicholas Sze
       
      16.
      Add a Namesystem interface to avoid directly referring to FSNamesystem Sub-task Closed Tsz Wo Nicholas Sze
       
      17.
      Move datanodes size printing to BlockManager from FSNameSystem's metasave API Sub-task Closed Uma Maheswara Rao G
       
      18.
      Remove reference to FSNamesystem in blockmanagement classes Sub-task Closed Tsz Wo Nicholas Sze
       
      19.
      Clean up FSNamesystem and BlockManager Sub-task Closed Tsz Wo Nicholas Sze
       
      20.
      change INode to package private Sub-task Resolved John George
       
      21.
      change INode to package private Sub-task Resolved John George
       
      22.
      blockmanagement should stop using INodeFile & INodeFileUC Sub-task Resolved John George
       
      23.
      change variable names referring to inode in blockmanagement to more appropriate Sub-task Closed John George
       
      24.
      Change INodeFile and INodeFileUnderConstruction to package private Sub-task Closed Tsz Wo Nicholas Sze
       
      25. The BlockManager should not use the FSN lock Sub-task Open Unassigned
       
      26. BlockId and Generation Stamp code should be moved to the blockmanagement package Sub-task Open Arpit Agarwal
       

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #710 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/710/)
        Fix CHANGES.txt (HDFS-2106 should be HDFS-2107) and add NumberReplicas.java.
        HDFS-2106. Move block management code from o.a.h.h.s.namenode to a new package o.a.h.h.s.blockmanagement.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1140913
        Files :

        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1140909
        Files :

        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hdfs/src/webapps/hdfs/block_info_xml.jsp
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/Host2NodesMap.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/CreateEditsLog.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestNamenodeCapacityReport.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCorruptReplicaInfo.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestHeartbeatHandling.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestReplicationPolicy.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestOverReplicatedBlocks.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestPendingReplication.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/net/TestNetworkTopology.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyDefault.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestCorruptReplicaInfo.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/DecommissionManager.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockInfoUnderConstruction.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicy.java
        • /hadoop/common/trunk/hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java
        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockManager.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestBlocksScheduledCounter.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/DatanodeDescriptor.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestComputeInvalidateWork.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDatanodeDescriptor.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/CorruptReplicasMap.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlocks.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestUnderReplicatedBlocks.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/UnderReplicatedBlocks.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestNodeCount.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/DFSTestUtil.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/PendingReplicationBlocks.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockInfo.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestHost2NodesMap.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #710 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/710/ ) Fix CHANGES.txt ( HDFS-2106 should be HDFS-2107 ) and add NumberReplicas.java. HDFS-2106 . Move block management code from o.a.h.h.s.namenode to a new package o.a.h.h.s.blockmanagement. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1140913 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1140909 Files : /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hdfs/src/webapps/hdfs/block_info_xml.jsp /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/Host2NodesMap.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/CreateEditsLog.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestNamenodeCapacityReport.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCorruptReplicaInfo.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestHeartbeatHandling.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestReplicationPolicy.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestOverReplicatedBlocks.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestPendingReplication.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/net/TestNetworkTopology.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/common/JspHelper.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestCorruptReplicaInfo.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/DecommissionManager.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockInfoUnderConstruction.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicy.java /hadoop/common/trunk/hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockManager.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestBlocksScheduledCounter.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/DatanodeDescriptor.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestComputeInvalidateWork.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDatanodeDescriptor.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/CorruptReplicasMap.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlocks.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestUnderReplicatedBlocks.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/UnderReplicatedBlocks.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestNodeCount.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/DFSTestUtil.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/PendingReplicationBlocks.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockInfo.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestHost2NodesMap.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #765 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/765/)
        Fix CHANGES.txt (HDFS-2106 should be HDFS-2107) and add NumberReplicas.java.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1140913
        Files :

        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #765 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/765/ ) Fix CHANGES.txt ( HDFS-2106 should be HDFS-2107 ) and add NumberReplicas.java. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1140913 Files : /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/NumberReplicas.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #764 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/764/)
        HDFS-2106. Move block management code from o.a.h.h.s.namenode to a new package o.a.h.h.s.blockmanagement.

        szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1140909
        Files :

        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
        • /hadoop/common/trunk/hdfs/src/webapps/hdfs/block_info_xml.jsp
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/Host2NodesMap.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/CreateEditsLog.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestNamenodeCapacityReport.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCorruptReplicaInfo.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestHeartbeatHandling.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestReplicationPolicy.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestOverReplicatedBlocks.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestPendingReplication.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/net/TestNetworkTopology.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyDefault.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestCorruptReplicaInfo.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/DecommissionManager.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockInfoUnderConstruction.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicy.java
        • /hadoop/common/trunk/hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java
        • /hadoop/common/trunk/hdfs/CHANGES.txt
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockManager.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestBlocksScheduledCounter.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/DatanodeDescriptor.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INode.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestComputeInvalidateWork.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDatanodeDescriptor.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/CorruptReplicasMap.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlocks.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestUnderReplicatedBlocks.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/UnderReplicatedBlocks.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestNodeCount.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/DFSTestUtil.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/PendingReplicationBlocks.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java
        • /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockInfo.java
        • /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestHost2NodesMap.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #764 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/764/ ) HDFS-2106 . Move block management code from o.a.h.h.s.namenode to a new package o.a.h.h.s.blockmanagement. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1140909 Files : /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java /hadoop/common/trunk/hdfs/src/webapps/hdfs/block_info_xml.jsp /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/Host2NodesMap.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/CreateEditsLog.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoUnderConstruction.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestHeartbeatHandling.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestNamenodeCapacityReport.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCorruptReplicaInfo.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestHeartbeatHandling.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestReplicationPolicy.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestOverReplicatedBlocks.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestPendingReplication.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/net/TestNetworkTopology.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/CorruptReplicasMap.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/common/JspHelper.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/UnderReplicatedBlocks.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeJspHelper.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INodeFile.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/protocol/BlockCommand.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestCorruptReplicaInfo.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/DecommissionManager.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockInfoUnderConstruction.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicy.java /hadoop/common/trunk/hdfs/src/test/unit/org/apache/hadoop/hdfs/server/namenode/TestNNLeaseRecovery.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java /hadoop/common/trunk/hdfs/CHANGES.txt /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockManager.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/TestBlocksScheduledCounter.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/DatanodeDescriptor.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/INode.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestComputeInvalidateWork.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDatanodeDescriptor.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/CorruptReplicasMap.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestUnderReplicatedBlocks.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestUnderReplicatedBlocks.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/UnderReplicatedBlocks.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestNodeCount.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/DFSTestUtil.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/PendingReplicationBlocks.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestBlockUnderConstruction.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/blockmanagement/TestDatanodeDescriptor.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/blockmanagement/PendingReplicationBlocks.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestDeadDatanode.java /hadoop/common/trunk/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BlockInfo.java /hadoop/common/trunk/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestHost2NodesMap.java
        Hide
        Tsz Wo Nicholas Sze added a comment -

        bm_api20110628.pdf: used dashed lines for heartbeat replies.

        Show
        Tsz Wo Nicholas Sze added a comment - bm_api20110628.pdf: used dashed lines for heartbeat replies.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        Hi Todd, thanks for taking a look. The operations probably can be done by heartbeat reply. I need to think about it in details.

        Show
        Tsz Wo Nicholas Sze added a comment - Hi Todd, thanks for taking a look. The operations probably can be done by heartbeat reply. I need to think about it in details.
        Hide
        Todd Lipcon added a comment -

        This stuff is very interesting!

        Do you have some fleshed out designs on how to handle some of the operations which are currently "transactional" between blocks and inodes? eg how is it handled when a pipeline is broken, and the block recovery has the side effect of changing an INodeFileUnderConstruction into an INodeFile? I don't see leases anywhere in the API diagram either.

        Show
        Todd Lipcon added a comment - This stuff is very interesting! Do you have some fleshed out designs on how to handle some of the operations which are currently "transactional" between blocks and inodes? eg how is it handled when a pipeline is broken, and the block recovery has the side effect of changing an INodeFileUnderConstruction into an INodeFile? I don't see leases anywhere in the API diagram either.
        Hide
        Tsz Wo Nicholas Sze added a comment -

        bm_api20110627.pdf: Block Management API

        Show
        Tsz Wo Nicholas Sze added a comment - bm_api20110627.pdf: Block Management API

          People

          • Assignee:
            Tsz Wo Nicholas Sze
            Reporter:
            Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            20 Start watching this issue

            Dates

            • Created:
              Updated:

              Development