Hadoop Common
  1. Hadoop Common
  2. HADOOP-681

Adminstrative hook to pull live nodes out of a HDFS cluster

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.10.0
    • Component/s: None
    • Labels:
      None

      Description

      Introduction
      ------------
      An administrator sometimes needs to bring down a datanode for scheduled maintenance. It would be nice if HDFS can be informed about this event. On receipt of this event, HDFS can take steps so that HDFS data is not lost when the node goes down at a later time.

      Architecture
      -----------
      In the existing architecture, a datanode can be in one of two states: dead or alive. A datanode is alive if its heartbeats are being processed by the namenode. Otherwise that datanode is in dead state. We extend the architecture to introduce the concept of a tranquil state for a datanode.
      A datanode is in tranquil state if:

      • it cannot be a target for replicating any blocks
      • any block replica that it currently contains does not count towards the target-replication-factor of that block

      Thus, a node that is in tranquil state can be brought down without impacting the guarantees provided by HDFS.

      The tranquil state is not persisted across namenode restarts. If the namenode restarts then that datanode will go back to being in the dead or alive state.

      The datanode is completely transparent to the fact that it has been labeled as being in tranquil state. It can continue to heartbeat and serve read requests for datablocks.

      DFSShell Design
      -----------------------
      We extend the DFS Shell utility to specify a list of nodes to the namenode.
      hadoop dfs -tranquil

      {set|clear|get}

      datanodename1 [,datanodename2]

      The DFSShell utility sends this list to the namenode. This DFSShell command invoked with the "set" option completes when the list is transferred to the namenode. This command is non-blocking; it returns before the datanode is actually in the tranquil state. The client can then query the state by re-issuing the command with the "get" option. This option will indicate whether the datanode is in tranquil state or is "being tranquiled". The "clear" option is used to transition a tranquil datanode to the alive state. The "clear" option is a no-op if the datanode is not in the "tranquil" state.

      ClientProtocol Design
      --------------------
      The ClientProtocol is the protocol exported by the namenode for its client.
      This protocol is extended to incorporate three new methods:
      ClientProtocol.setTranquil(String[] datanodes)
      ClientProtocol.getTranquil(String datanode)
      ClientProtocol.clearTranquil(String[] datanodes)

      The ProtocolVersion is incremented to prevent conversations between imcompatible clients and servers. An old DFSShell cannot talk to the new NameNode and vice-versa.

      NameNode Design
      -------------------------
      The namenode does the bulk of the work for supporting this new feature.

      The DatanodeInfo object has a new private member named "state". It also has three new member functions:
      datanodeInfo.tranquilStarted(): start the process of tranquilization
      datanodeInfo.tranquilCompleted(): node is not in tranquil state
      datanodeInfo.clearTranquil() : remove tranquilization from node

      The namenode exposes a new API to set and clear tranquil states for a datanode. On receipt of a "set tranquil" command, it invokes datanodeInfo.tranquilStarted().

      The FSNamesystem.chooseTarget() method skips over datanodes that are marked as being in the "tranquil" state. This ensures that tranquil-datanodes are never chosen as targets of replication. The namenode does not record
      this operation in either the FsImage or the EditLogs.

      The namenode puts all the blocks from a being-tranquiled node into the neededReplication data structure. Necessary code changes are made to ensure that these blocks get replicated by the regular replication method. As of now, the regular replication code does not distinguish between these blocks and the blocks that are replication candidates because some other datanode might have died. It might be prudent to give different (lower?) weightage to this type of replication requests, but that exercise is deferred to a later date. In this design, replication requests generated because of a node going to a tranquil state are not distinguished from replication requests generated by a datanode going to the dead state.

      The DatanodeInfo object has another new private member named "pendingTranquilCount". This field stores the remaining number of blocks that still remain to be replicated. This field is valid only if the node is in the ets being-tranquiled state. On receipt of every 'n' heartbeats from the being-tranquiled datanode, the namenode calculates the amount of data that is still remaining to be replicated and updates the "pendingTranquilCount". in the DatanodeInfo.When all the replications complete, the datanode is marked as tranquiled. The number 'n' is selected in such a way that the average heartbeat processing time does not increase appreciably.

      It is possible that the namenode might stop receving heartbeats from a datanode that is being-tranquiled. In this case, the tranquil flag of the datanode gets cleared. It transitions to the dead state and the normal processing for alive-to-dead transition occurs here.

      Web Interface
      -------------------
      The dfshealth.jsp displays the live nodes, dead nodes, being-tranquiled and tranquil nodes. For nodes in the being-tranquiled state, it displays the percentage of tranquilization completed till now.

      Issues
      --------
      1. If a request for tranquilization starts getting processed and there aren't enough space available in DFS to complete the necessary replication, then that node might remain in the being-tranquiled state for a long long time. This is not necessarily a bad thing but is there a better option?

      2. We have opted for not storing cluster configuration information in the persistent image of the file system. (The tranquil state of a datanode may be lost if the namenode restarts).

      1. nodedecommission5.patch
        40 kB
        dhruba borthakur

        Activity

        dhruba borthakur created issue -
        Hide
        eric baldeschwieler added a comment -

        Looks good overall. Some comments on the details...

        The tranquil state should be persistent IMO. This should be recorded to disk and backed up anywhere we backup changelog data.

        the dfsshell command is for user commands, admin commands should have a distinct API to avoid confusion allow later divergence in security / etc.

        I'd prefer it if remote tools could not be used without changing the state on the namenode to effect its operation. This is a simple way of proving that the reconfig is valid. For example, change a file listing the idled nodes and then ping the server to reparse it.

        HDFS is not stable without a stable definition of the cluster, we need to stop building APIs that ignore this.

        BTW tranquil is a little obscure. What about idle or decommissioned or toBeKilled?

        Show
        eric baldeschwieler added a comment - Looks good overall. Some comments on the details... The tranquil state should be persistent IMO. This should be recorded to disk and backed up anywhere we backup changelog data. the dfsshell command is for user commands, admin commands should have a distinct API to avoid confusion allow later divergence in security / etc. I'd prefer it if remote tools could not be used without changing the state on the namenode to effect its operation. This is a simple way of proving that the reconfig is valid. For example, change a file listing the idled nodes and then ping the server to reparse it. HDFS is not stable without a stable definition of the cluster, we need to stop building APIs that ignore this. BTW tranquil is a little obscure. What about idle or decommissioned or toBeKilled?
        Hide
        Konstantin Shvachko added a comment -

        1. I'd replace 3 new methods in ClientProtocol by one with a 3 valued parameter.
        2. You do not need a new member for counting already re-replicated blocks.
        You can count blocks in the DatanodeDescriptor.
        When the set of blocks is empty replication is finished.
        3. Name-node can send shutdown command to the data-node after that
        (in reply to the heartbeat). The command is implemented but has not been used yet.
        4. Inside name-node DatanodeDescriptor should be used instead of DatanodeInfo.

        Show
        Konstantin Shvachko added a comment - 1. I'd replace 3 new methods in ClientProtocol by one with a 3 valued parameter. 2. You do not need a new member for counting already re-replicated blocks. You can count blocks in the DatanodeDescriptor. When the set of blocks is empty replication is finished. 3. Name-node can send shutdown command to the data-node after that (in reply to the heartbeat). The command is implemented but has not been used yet. 4. Inside name-node DatanodeDescriptor should be used instead of DatanodeInfo.
        Hide
        Yoram Arnon added a comment -

        +1 to Eric's comments.

        in particular, dfs -report, dfs -safemode and dfs -decommision should be separated into a dfsadmin command.

        And, an API to remove datanodes from the namenode's eternal memory would be good in general, and useful in this case

        Show
        Yoram Arnon added a comment - +1 to Eric's comments. in particular, dfs -report, dfs -safemode and dfs -decommision should be separated into a dfsadmin command. And, an API to remove datanodes from the namenode's eternal memory would be good in general, and useful in this case
        Hide
        dhruba borthakur added a comment -

        Thanks a bunch: Eric, Yoram & Kons for your comments. Here are my take-aways:

        1. I will change the name of this new state from tranquil to decommission.

        2. I will make the "decommission" a persistent state. This will be done in a follow-on patch submission (after the disk format upgrade patch) is incorporated.

        3. I will create a new command called dfsadmin. It will be invoked as
        bin/hadoop --config conf dfsadmin -decommision

        4. I will incorprate most of Konstantin's comments. However, the namenode will not send shutdown command to the datanode after all replications are completed. In fact, we endeavour to keep that decommissioned datanode alive and use it (if needed) to serve read requests.

        4. I will create a separate defect to move "dfs -report" and "dfs -safemode" to the dfsadmin command. I will create another separate defect to support "dfsadmin -removenode" to completely remove a datanodes's information from the namenode.

        Show
        dhruba borthakur added a comment - Thanks a bunch: Eric, Yoram & Kons for your comments. Here are my take-aways: 1. I will change the name of this new state from tranquil to decommission. 2. I will make the "decommission" a persistent state. This will be done in a follow-on patch submission (after the disk format upgrade patch) is incorporated. 3. I will create a new command called dfsadmin. It will be invoked as bin/hadoop --config conf dfsadmin -decommision 4. I will incorprate most of Konstantin's comments. However, the namenode will not send shutdown command to the datanode after all replications are completed. In fact, we endeavour to keep that decommissioned datanode alive and use it (if needed) to serve read requests. 4. I will create a separate defect to move "dfs -report" and "dfs -safemode" to the dfsadmin command. I will create another separate defect to support "dfsadmin -removenode" to completely remove a datanodes's information from the namenode.
        Hide
        dhruba borthakur added a comment -

        This is the first version of code change. Review comments welcome.

        Show
        dhruba borthakur added a comment - This is the first version of code change. Review comments welcome.
        dhruba borthakur made changes -
        Field Original Value New Value
        Attachment nodedecommission1.patch [ 12345792 ]
        Hide
        dhruba borthakur added a comment -

        The previous patch went stale, it was unable to automatic merge. Uploading a more recent patch file for code review.

        Show
        dhruba borthakur added a comment - The previous patch went stale, it was unable to automatic merge. Uploading a more recent patch file for code review.
        dhruba borthakur made changes -
        Attachment nodedecommission2.patch [ 12345991 ]
        dhruba borthakur made changes -
        Attachment nodedecommission1.patch [ 12345792 ]
        Hide
        Konstantin Shvachko added a comment -

        Comments:

        = Index: src/java/org/apache/hadoop/dfs/ClientProtocol.java
        public static final long versionID = 4L;
        You should write a comment what exactly changed for the ClientProtocol in new version (compared to the previous one).
        That way you will be able to track changes back from version to version looking at the history of changes.

        = Index: src/java/org/apache/hadoop/dfs/DFSClient.java
        may be it is better to use DatanodeID instead of mere String as a node parameter in decommission methods;
        this will require an additional public constructor DatanodeID( String nodeName )

        = Index: src/java/org/apache/hadoop/dfs/DFSAdmin.java
        decommission() does not document the return value
        boolean mode is never used
        final String safeModeUsage is never used
        decommission usage does not specify data-node parameters,
        it is not clear what identifies data-nodes (host-port or just host, or storage id)

        = Index: src/java/org/apache/hadoop/dfs/FSNamesystem.java
        decommissionInProgress() should start with is***()
        replicationInProgress() should start with is***()
        in startDecommission() and stopDecommission() it is better to call public method getName()
        rather than directly accessing node.name the protected member
        Block decommissionblocks[] should be decommissionBlocks, please check other places too.
        May be it is not a part of this patch but at some point we should combine two methods datanodeReport() and
        DFSNodesStatus() so that data-node reports for the webUI and DFSShell were the same.

        = Index: src/java/org/apache/hadoop/dfs/DatanodeDescriptor.java
        decommissioned() should start with is***()
        I don't think these constants are used anywhere in the code. They could be confused with the enum values having the same names.
        public static final int NORMAL = 0;
        public static final int DECOMMISSION_INPROGRESS = 1;
        public static final int DECOMMISSIONED = 2;
        I propose to rename AdminStates to DecommissionState and eliminate NORMAL state replacing it by null, where applicable.
        with a clear (imo) semantics: no decommission - no state.
        setAdminState() never uses the return value
        setAdminState( DECOMMISSIONED ) and setDecommissioned() are 2 ways to do the same thing

        = Index: src/java/org/apache/hadoop/dfs/DatanodeInfo.java
        = Index: src/java/org/apache/hadoop/dfs/DatanodeReport.java
        I don't think this class should be introduced at all.
        DatanodeReport effectively returns an entire DatanodeDescriptor.
        The original design of DatanodeInfo and DatanodeDescriptor distinguishes them in the way, that
        DatanodeDescriptor is an internal name-node class which is never passed to the outside world.
        Anything that should be returned back to a client or a data-node should be contained in the DatanodeInfo.
        So it is better to divide decommission-related methods and members to those that are name-node specific
        and those that are visible to the clients. The former should remain in the DatanodeDescriptor, and the latter
        should go into DatanodeInfo. E.g.
        DatanodeInfo has private decommissionState and getters isDecommissioned() and isDecommissionInProgress()
        as well as getDatanodeReport(),
        everything else is in DatanodeDescriptor.

        Show
        Konstantin Shvachko added a comment - Comments: = Index: src/java/org/apache/hadoop/dfs/ClientProtocol.java public static final long versionID = 4L; You should write a comment what exactly changed for the ClientProtocol in new version (compared to the previous one). That way you will be able to track changes back from version to version looking at the history of changes. = Index: src/java/org/apache/hadoop/dfs/DFSClient.java may be it is better to use DatanodeID instead of mere String as a node parameter in decommission methods; this will require an additional public constructor DatanodeID( String nodeName ) = Index: src/java/org/apache/hadoop/dfs/DFSAdmin.java decommission() does not document the return value boolean mode is never used final String safeModeUsage is never used decommission usage does not specify data-node parameters, it is not clear what identifies data-nodes (host-port or just host, or storage id) = Index: src/java/org/apache/hadoop/dfs/FSNamesystem.java decommissionInProgress() should start with is***() replicationInProgress() should start with is***() in startDecommission() and stopDecommission() it is better to call public method getName() rather than directly accessing node.name the protected member Block decommissionblocks[] should be decommissionBlocks, please check other places too. May be it is not a part of this patch but at some point we should combine two methods datanodeReport() and DFSNodesStatus() so that data-node reports for the webUI and DFSShell were the same. = Index: src/java/org/apache/hadoop/dfs/DatanodeDescriptor.java decommissioned() should start with is***() I don't think these constants are used anywhere in the code. They could be confused with the enum values having the same names. public static final int NORMAL = 0; public static final int DECOMMISSION_INPROGRESS = 1; public static final int DECOMMISSIONED = 2; I propose to rename AdminStates to DecommissionState and eliminate NORMAL state replacing it by null, where applicable. with a clear (imo) semantics: no decommission - no state. setAdminState() never uses the return value setAdminState( DECOMMISSIONED ) and setDecommissioned() are 2 ways to do the same thing = Index: src/java/org/apache/hadoop/dfs/DatanodeInfo.java = Index: src/java/org/apache/hadoop/dfs/DatanodeReport.java I don't think this class should be introduced at all. DatanodeReport effectively returns an entire DatanodeDescriptor. The original design of DatanodeInfo and DatanodeDescriptor distinguishes them in the way, that DatanodeDescriptor is an internal name-node class which is never passed to the outside world. Anything that should be returned back to a client or a data-node should be contained in the DatanodeInfo. So it is better to divide decommission-related methods and members to those that are name-node specific and those that are visible to the clients. The former should remain in the DatanodeDescriptor, and the latter should go into DatanodeInfo. E.g. DatanodeInfo has private decommissionState and getters isDecommissioned() and isDecommissionInProgress() as well as getDatanodeReport(), everything else is in DatanodeDescriptor.
        Hide
        dhruba borthakur added a comment -

        Hi Konstantin,

        Thanks for your comments. My comments marked with <****>

        1. Index: src/java/org/apache/hadoop/dfs/ClientProtocol.java
        public static final long versionID = 4L; You should write a comment
        <****> Done.

        2. Index: src/java/org/apache/hadoop/dfs/DFSClient.java
        may be it is better to use DatanodeID instead of mere String
        <****>The user can specify a name or a name:port. The code will match both formats. That's the reason
        the user input is accepted as a string rather than a DatanodeID.

        3. Index: src/java/org/apache/hadoop/dfs/DFSAdmin.java
        decommission() does not document the return value; boolean mode is never used
        final String safeModeUsage is never used decommission usage does not specify data-node parameters,
        <****> Done

        4. Index: src/java/org/apache/hadoop/dfs/FSNamesystem.java
        decommissionInProgress() should start with is**(); replicationInProgress() should start with is**()
        in startDecommission() and stopDecommission() it is better to call public method getName()
        Block decommissionblocks[] should be decommissionBlocks.
        <****> Done

        5. Index: src/java/org/apache/hadoop/dfs/DatanodeDescriptor.java
        decommissioned() should start with is***()
        I don't think these constants are used anywhere in the code. They could be confused with the enum
        values having the same names. public static final int NORMAL = 0;
        <****> Done

        6. I propose to rename AdminStates to DecommissionState and eliminate NORMAL state replacing it by null,
        where applicable. with a clear (imo) semantics: no decommission - no state.
        <****> I have, on purpose, kept the API to be a little more generic. In the future, there could be more
        adminstrative states of datanodes, e.g. ReadOnly Datanodes

        7. Index: src/java/org/apache/hadoop/dfs/DatanodeReport.java
        I don't think this class should be introduced at all.
        DatanodeReport effectively returns an entire DatanodeDescriptor.
        <****> There is one subtle difference between DatanodeReport and DatanodeInfo: their serialization
        methods. The Namenode serializes DatanodeInfo while writing it to the FsImage. I do not want
        any FSImage format change at present. The persistentcy of adminState will be bundled in with
        other FsImage changes at a later date. The DatanodeReport class has to report the
        adminState to the UI, thus it has to serialize the adminState too.

        Please let me know if my counter-proposals sound ok.

        Show
        dhruba borthakur added a comment - Hi Konstantin, Thanks for your comments. My comments marked with <****> 1. Index: src/java/org/apache/hadoop/dfs/ClientProtocol.java public static final long versionID = 4L; You should write a comment <****> Done. 2. Index: src/java/org/apache/hadoop/dfs/DFSClient.java may be it is better to use DatanodeID instead of mere String <****>The user can specify a name or a name:port. The code will match both formats. That's the reason the user input is accepted as a string rather than a DatanodeID. 3. Index: src/java/org/apache/hadoop/dfs/DFSAdmin.java decommission() does not document the return value; boolean mode is never used final String safeModeUsage is never used decommission usage does not specify data-node parameters, <****> Done 4. Index: src/java/org/apache/hadoop/dfs/FSNamesystem.java decommissionInProgress() should start with is** (); replicationInProgress() should start with is **() in startDecommission() and stopDecommission() it is better to call public method getName() Block decommissionblocks[] should be decommissionBlocks. <****> Done 5. Index: src/java/org/apache/hadoop/dfs/DatanodeDescriptor.java decommissioned() should start with is***() I don't think these constants are used anywhere in the code. They could be confused with the enum values having the same names. public static final int NORMAL = 0; <****> Done 6. I propose to rename AdminStates to DecommissionState and eliminate NORMAL state replacing it by null, where applicable. with a clear (imo) semantics: no decommission - no state. <****> I have, on purpose, kept the API to be a little more generic. In the future, there could be more adminstrative states of datanodes, e.g. ReadOnly Datanodes 7. Index: src/java/org/apache/hadoop/dfs/DatanodeReport.java I don't think this class should be introduced at all. DatanodeReport effectively returns an entire DatanodeDescriptor. <****> There is one subtle difference between DatanodeReport and DatanodeInfo: their serialization methods. The Namenode serializes DatanodeInfo while writing it to the FsImage. I do not want any FSImage format change at present. The persistentcy of adminState will be bundled in with other FsImage changes at a later date. The DatanodeReport class has to report the adminState to the UI, thus it has to serialize the adminState too. Please let me know if my counter-proposals sound ok.
        Hide
        Konstantin Shvachko added a comment -

        6. From the top of my mind I cannot think of any other use of admin states right now.
        My intention was to minimize the impact on the system state when decommissions are not happening.

        7. I thought the intention was to make the decommission state persistent.
        If not then just reset it to NORMAL or null while loading the image.
        I don't think we should introduce a new class for temporary purposes.

        May be we should keep DatanodeInfo unchanged and implement a separate call
        getDecommissionedNodes( inProgress, decommissioned );
        The web UI is getting the DatanodeDescriptor anyway now.

        Show
        Konstantin Shvachko added a comment - 6. From the top of my mind I cannot think of any other use of admin states right now. My intention was to minimize the impact on the system state when decommissions are not happening. 7. I thought the intention was to make the decommission state persistent. If not then just reset it to NORMAL or null while loading the image. I don't think we should introduce a new class for temporary purposes. May be we should keep DatanodeInfo unchanged and implement a separate call getDecommissionedNodes( inProgress, decommissioned ); The web UI is getting the DatanodeDescriptor anyway now.
        Hide
        dhruba borthakur added a comment -

        6. I would prefer to keep the name "adminstate" unless you strongly insist otherwise. There isn't any impact to the system when no nodes are being decommissioned.

        7. The DatanodeDescriptor is persisted in the fsimage whereas DatanodeReport is an object used by the ClientProtocol. In theory, I would like to keep them separate. The first one is an on-disk structure whereas the second one is a over-the-wire-protocol structure. In fact, at a later date, I was planning on changing the webUI to use Datanode Report.

        My theory is to keep two different object for two different purposes, especially because their serialization requirements are different. The first oject DatanodeDescriptor gets written to disk while the second one DatanodeReport gets passed back to the client. In the DatanodeReport, I can visualize that other parameters will be added in the future, e.g. we could be adding computed statistics into the Report. These computed statistics might have to computed using a variety of namenode-data structures (instead of just using DatanodeDescriptor).

        Show
        dhruba borthakur added a comment - 6. I would prefer to keep the name "adminstate" unless you strongly insist otherwise. There isn't any impact to the system when no nodes are being decommissioned. 7. The DatanodeDescriptor is persisted in the fsimage whereas DatanodeReport is an object used by the ClientProtocol. In theory, I would like to keep them separate. The first one is an on-disk structure whereas the second one is a over-the-wire-protocol structure. In fact, at a later date, I was planning on changing the webUI to use Datanode Report. My theory is to keep two different object for two different purposes, especially because their serialization requirements are different. The first oject DatanodeDescriptor gets written to disk while the second one DatanodeReport gets passed back to the client. In the DatanodeReport, I can visualize that other parameters will be added in the future, e.g. we could be adding computed statistics into the Report. These computed statistics might have to computed using a variety of namenode-data structures (instead of just using DatanodeDescriptor).
        Hide
        Konstantin Shvachko added a comment -

        6. The name is not so important. I'm concerned about the NORMAL state.
        You will have to store an object for each data-node, which for most of them
        will have the same value NORMAL. Treating null as the normal state would
        be more space efficient.

        7. I am just saying there is already an over-the-wire class. It is called DatanodeInfo.
        The distinction between DatanodeDescriptor and DatanodeInfo is exactly as you described for the DatanodeReport.
        DatanodeReport = DatanodeInfo?

        There is a problem with serialization of DatanodeDescriptor, since it is not versioned.
        Version number is required if fields are changing.
        Versioning was discussed in HADOOP-224, but was never resolved.
        A simple solution I'd propose is to create a new interface
        Storable

        { readObject( version, inStream ) throws IOException; writeObject( version, outStream ) throws IOException; }

        Storable is used instead of Writable whenever an object needs to be stored on disk.
        Writable is used solely for communication related serialization.

        Show
        Konstantin Shvachko added a comment - 6. The name is not so important. I'm concerned about the NORMAL state. You will have to store an object for each data-node, which for most of them will have the same value NORMAL. Treating null as the normal state would be more space efficient. 7. I am just saying there is already an over-the-wire class. It is called DatanodeInfo. The distinction between DatanodeDescriptor and DatanodeInfo is exactly as you described for the DatanodeReport. DatanodeReport = DatanodeInfo? There is a problem with serialization of DatanodeDescriptor, since it is not versioned. Version number is required if fields are changing. Versioning was discussed in HADOOP-224 , but was never resolved. A simple solution I'd propose is to create a new interface Storable { readObject( version, inStream ) throws IOException; writeObject( version, outStream ) throws IOException; } Storable is used instead of Writable whenever an object needs to be stored on disk. Writable is used solely for communication related serialization.
        Hide
        dhruba borthakur added a comment -

        Hi folks,

        I would really like some comments on Konstantin's proposal to introduce the Storable Interface to persist namenode metadata on disk. This is to support versioned-persistent-metadata for HDFS.

        thanks,
        dhruba

        Show
        dhruba borthakur added a comment - Hi folks, I would really like some comments on Konstantin's proposal to introduce the Storable Interface to persist namenode metadata on disk. This is to support versioned-persistent-metadata for HDFS. thanks, dhruba
        dhruba borthakur made changes -
        Attachment nodedecommission2.patch [ 12345991 ]
        Hide
        dhruba borthakur added a comment -

        Code changes after incorporating Code review comments. The only outstanding issue left is the issue of changing the fsImage format (to persistent decommissioned state).

        Show
        dhruba borthakur added a comment - Code changes after incorporating Code review comments. The only outstanding issue left is the issue of changing the fsImage format (to persistent decommissioned state).
        dhruba borthakur made changes -
        Attachment nodedecommission3.patch [ 12346316 ]
        dhruba borthakur made changes -
        Attachment nodedecommission3.patch [ 12346316 ]
        dhruba borthakur made changes -
        Attachment nodedecommission5.patch [ 12348175 ]
        Hide
        dhruba borthakur added a comment -

        Code for decommissioning nodes. All review comments (thanks Konstantin) have been incorporated.

        Show
        dhruba borthakur added a comment - Code for decommissioning nodes. All review comments (thanks Konstantin) have been incorporated.
        dhruba borthakur made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Hadoop QA added a comment -

        +1, because http://issues.apache.org/jira/secure/attachment/12348175/nodedecommission5.patch applied and successfully tested against trunk revision r489707.

        Show
        Hadoop QA added a comment - +1, because http://issues.apache.org/jira/secure/attachment/12348175/nodedecommission5.patch applied and successfully tested against trunk revision r489707.
        Hide
        Doug Cutting added a comment -

        I just committed this. Thanks, Dhruba!

        Show
        Doug Cutting added a comment - I just committed this. Thanks, Dhruba!
        Doug Cutting made changes -
        Fix Version/s 0.10.0 [ 12312207 ]
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Doug Cutting made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Owen O'Malley made changes -
        Component/s dfs [ 12310710 ]

          People

          • Assignee:
            dhruba borthakur
            Reporter:
            dhruba borthakur
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development