[HADOOP-5257] Export namenode/datanode functionality through a pluggable RPC layer

• Type: New Feature
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 0.21.0
• Hadoop Flags: Reviewed
• Release Note: New plugin facility for namenode and datanode instantiates classes named in new configuration properties dfs.datanode.plugins and dfs.namenode.plugins.
• Assignee: Carlos Valiente
• Reporter: Carlos Valiente
• Created: 13/Feb/09 18:42
• Updated: 06/May/13 04:56
• Resolved: 07/Apr/09 19:12

Description

Adding support for pluggable components would allow exporting DFS functionallity using arbitrary protocols, like Thirft or Protocol Buffers. I'm opening this issue on Dhruba's suggestion in HADOOP-4707.

Plug-in implementations would extend this base class:

abstract class Plugin {

    public abstract datanodeStarted(DataNode datanode);

    public abstract datanodeStopping();

    public abstract namenodeStarted(NameNode namenode);

    public abstract namenodeStopping();
}

Name node instances would then start the plug-ins according to a configuration object, and would also shut them down when the node goes down:

public class NameNode {

    // [..]

    private void initialize(Configuration conf)
        // [...]
        for (Plugin p: PluginManager.loadPlugins(conf))
          p.namenodeStarted(this);
    }

    // [..]

    public void stop() {
        if (stopRequested)
            return;
        stopRequested = true;
        for (Plugin p: plugins) 
            p.namenodeStopping();
        // [..]
    }

    // [..]
}

Data nodes would do a similar thing in DataNode.startDatanode() and DataNode.shutdown

Issue Links

• blocks: HDFS-417 Improvements to Hadoop Thrift bindings
• is related to: HADOOP-8832 backport serviceplugin to branch-1
• is related to: HDFS-3963 backport namenode/datanode serviceplugin to branch-1
• relates to: HDFS-217 Need an FTP Server implementation over HDFS
• relates to: MAPREDUCE-461 Enable ServicePlugins for the JobTracker