[HDFS-6994] libhdfs3 - A native C/C++ HDFS client

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: hdfs-client
• Assignee: Zhanwei Wang
• Reporter: Zhanwei Wang
• Created: 04/Sep/14 06:39
• Updated: 21/Jun/23 10:59

Description

Hi All

I just got the permission to open source libhdfs3, which is a native C/C++ HDFS client based on Hadoop RPC protocol and HDFS Data Transfer Protocol.

libhdfs3 provide the libhdfs style C interface and a C++ interface. Support both HADOOP RPC version 8 and 9. Support Namenode HA and Kerberos authentication.

libhdfs3 is currently used by Apache HAWQ at: https://github.com/apache/hawq/tree/master/depends/libhdfs3

I'd like to integrate libhdfs3 into HDFS source code to benefit others.

The libhdfs3 code originally from Pivotal was available on github at:
https://github.com/Pivotal-Data-Attic/pivotalrd-libhdfs3
http://pivotal-data-attic.github.io/pivotalrd-libhdfs3/

Issue Links

• is related to: HADOOP-10388 Pure native hadoop client

Sub-Tasks

• Add unit test for libhdfs3: Open
• Add function test for C interface, filesystem, InputStream and OutputStream for libhdfs3: Open
• Add function test for secure enabled HDFS for libhdfs3: Open
• Remove usage of boost::atomic in libhdfs3 to use old version of boost: Open
• Rename libhdfs3 to libndfs++: In Progress
• Revisit the public API of the libraries: Open
• Test libhdfs3 against MiniDFSCluster: In Progress
• Implement packet memory pool in output stream in libhdfs3: In Progress
• libhdfs3: fix strerror_r detection: Open
• Get HA NameNode information from config file: Open
• Revisit Windows socket API compatibility: Open
• Revisit the use of long data type: Open
• remove libuuid dependency: Open