[HADOOP-10388] Pure native hadoop client

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: HADOOP-10388
• Target Version/s: HADOOP-10388
• Assignee: Colin McCabe
• Reporter: Binglin Chang
• Created: 06/Mar/14 04:08
• Updated: 08/Sep/17 02:14

Description

A pure native hadoop client has following use case/advantages:
1. writing Yarn applications using c++
2. direct access to HDFS, without extra proxy overhead, comparing to web/nfs interface.
3. wrap native library to support more languages, e.g. python
4. lightweight, small footprint compare to several hundred MB of JDK and hadoop library with various dependencies.

Issue Links

• relates to: HDFS-6994 libhdfs3 - A native C/C++ HDFS client

Sub-Tasks

• Implement DataTransferProtocol in libhdfs-core.so: Open
• Pure Native Client: implement C code native_mini for YARN for unit test: Open
• native client: parse Hadoop permission strings: Open
• hconf.c: fix bug where we would sometimes not try to load multiple XML files from the same path: Patch Available
• implement ndfs_get_hosts: In Progress