[HDFS-417] Improvements to Hadoop Thrift bindings

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix

Environment

Tested under Linux x86-64

• Assignee: Todd Lipcon
• Reporter: Carlos Valiente
• Created: 21/Nov/08 21:35
• Updated: 18/May/11 21:48
• Resolved: 15/May/10 05:28

Description

I have made the following changes to hadoopfs.thrift:

1. Added namespaces for Python, Perl and C++.

1. Renamed parameters and struct members to camelCase versions to keep them consistent (in particular FileStatus {blockReplication,blockSize}

   vs FileStatus.

   {block_replication,blocksize}

   ).

1. Renamed ThriftHadoopFileSystem to FileSystem. From the perspective of a Perl/Python/C++ user, 1) it is already clear that we're using Thrift, and 2) the fact that we're dealing with Hadoop is already explicit in the namespace. The usage of generated code is more compact and (in my opinion) clearer:

   Perl:
   use HadoopFS;

   my $client = HadoopFS::FileSystemClient->new(..);

   instead of:

   my $client = HadoopFS::ThriftHadoopFileSystemClient->new(..);

   Python:

   from hadoopfs import FileSystem

   client = FileSystem.Client(..)

   instead of

   from hadoopfs import ThriftHadoopFileSystem

   client = ThriftHadoopFileSystem.Client(..)

   (See also the attached diff [^scripts_hdfs_py.diff] for the
   new version of 'scripts/hdfs.py').

   C++:

   hadoopfs::FileSystemClient client(..);

   instead of:

   hadoopfs::ThriftHadoopFileSystemClient client(..);

1. Renamed ThriftHandle to FileHandle: As in 3, it is clear that we're dealing with a Thrift object, and its purpose (to act as a handle for file operations) is clearer.

1. Renamed ThriftIOException to IOException, to keep it simpler, and consistent with MalformedInputException.

1. Added explicit version tags to fields of ThriftHandle/FileHandle, Pathname, MalformedInputException and ThriftIOException/IOException, to improve compatibility of existing clients with future versions of the interface which might add new fields to those objects (like stack traces for the exception types, for instance).

Those changes are reflected in the attachment hadoopfs_thrift.diff.

Changes in generated Java, Python, Perl and C++ code are also attached in gen.diff. They were generated by a Thrift checkout from trunk
(http://svn.apache.org/repos/asf/incubator/thrift/trunk/) as of revision
719697, plus the following Perl-related patches:

• https://issues.apache.org/jira/browse/THRIFT-190
• https://issues.apache.org/jira/browse/THRIFT-193
• https://issues.apache.org/jira/browse/THRIFT-199

The Thrift jar file libthrift.jar built from that Thrift checkout is also attached, since it's needed to run the Java Thrift server.

I have also added a new target to src/contrib/thriftfs/build.xml to build the Java bindings needed for org.apache.hadoop.thriftfs.HadoopThriftServer.java (see attachment build_xml.diff and modified HadoopThriftServer.java to make use of the new bindings (see attachment HadoopThriftServer_java.diff).

The jar file [^lib/hadoopthriftapi.jar] is also included, although it can be regenerated from the stuff under 'gen-java' and the new 'compile-gen' Ant target.

The whole changeset is also included as all.diff.

Issue Links

• is blocked by: HADOOP-5257 Export namenode/datanode functionality through a pluggable RPC layer
• is blocked by: HADOOP-5640 Allow ServicePlugins to hook callbacks into key service events
• is related to: HADOOP-5640 Allow ServicePlugins to hook callbacks into key service events