Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6994

libhdfs3 - A native C/C++ HDFS client

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • hdfs-client
    • None

    Description

      Hi All

      I just got the permission to open source libhdfs3, which is a native C/C++ HDFS client based on Hadoop RPC protocol and HDFS Data Transfer Protocol.

      libhdfs3 provide the libhdfs style C interface and a C++ interface. Support both HADOOP RPC version 8 and 9. Support Namenode HA and Kerberos authentication.

      libhdfs3 is currently used by Apache HAWQ at: https://github.com/apache/hawq/tree/master/depends/libhdfs3

      I'd like to integrate libhdfs3 into HDFS source code to benefit others.

      The libhdfs3 code originally from Pivotal was available on github at:
      https://github.com/Pivotal-Data-Attic/pivotalrd-libhdfs3
      http://pivotal-data-attic.github.io/pivotalrd-libhdfs3/

      Attachments

        1. HDFS-6994.patch
          3.46 MB
          Zhanwei Wang
        2. HDFS-6994-rpc-8.patch
          3.41 MB
          Zhanwei Wang

        Issue Links

          is related to

          New Feature - A new feature of the product, which has yet to be developed. HADOOP-10388 Pure native hadoop client

          • Major - Major loss of function.
          • Open
          1.
          		 boot up libhdfs3 project Sub-task Resolved Colin McCabe
          2.
          		 Implement basic utilities for libhdfs3 Sub-task Resolved Colin McCabe
          3.
          		 Add hdfs native client RPC functionality Sub-task Resolved Colin McCabe
          4.
          		 Implement RPC framework version 9 for libhdfs3 Sub-task Resolved Unassigned
          5.
          		 Use protobuf files in Hadoop source tree instead of the copy in libhdfs3 Sub-task Resolved Unassigned
          6.
          		 Implement input streams and file system functionality Sub-task Resolved Zhanwei Wang
          7.
          		 Implement C++ interface for file system for libhdfs3 Sub-task Resolved Unassigned
          8.
          		 Implement OutputStream for libhdfs3 Sub-task Resolved Zhanwei Wang
          9.
          		 Implement hdfs.h interface in libhdfs3 Sub-task Resolved Zhanwei Wang
          10.
          		 Implement DataTransferProtocol and Inputstream for libhdfs3 Sub-task Resolved Unassigned
          11.
          		 Add unit test for libhdfs3 Sub-task Open Unassigned
          12.
          		 Add function test for C interface, filesystem, InputStream and OutputStream for libhdfs3 Sub-task Open Unassigned
          13.
          		 Add function test for secure enabled HDFS for libhdfs3 Sub-task Open Unassigned
          14.
          		 use libexpat instead of libxml2 for libhdfs3 Sub-task Resolved Colin McCabe
          15.
          		 Remove usage of boost::atomic in libhdfs3 to use old version of boost Sub-task Open Unassigned
          16.
          		 support build libhdfs3 on windows Sub-task Resolved Thanh Do
          17.
          		 Rename libhdfs3 to libndfs++ Sub-task In Progress Colin McCabe
          18.
          		 Revisit the public API of the libraries Sub-task Open Haohui Mai
          19.
          		 Test libhdfs3 against MiniDFSCluster Sub-task In Progress Zhanwei Wang
          20.
          		 Implement packet memory pool in output stream in libhdfs3 Sub-task In Progress Zhanwei Wang
          21.
          		 Fix Atoi.cc link error Sub-task Resolved Unassigned
          22.
          		 Make cmake work in Windows Visual Studio 2010 Sub-task Resolved Thanh Do
          23.
          		 Add additional headers that includes need by Windows Sub-task Resolved Thanh Do
          24.
          		 Break the dependency between libnative_mini_dfs and libhdfs Sub-task Closed Zhanwei Wang
          25.
          		 Separate Platform specific funtions Sub-task Resolved Thanh Do
          26.
          		 libhdfs3: fix strerror_r detection Sub-task Open Thanh Do
          27.
          		 Get HA NameNode information from config file Sub-task Open Unassigned
          28.
          		 Revisit Windows socket API compatibility Sub-task Open Unassigned
          29.
          		 Revisit the use of long data type Sub-task Open Unassigned
          30.
          		 remove libuuid dependency Sub-task Open Unassigned

          Activity

            People

              wangzw Zhanwei Wang
              wangzw Zhanwei Wang
              Votes:
              5 Vote for this issue
              Watchers:
              49 Start watching this issue

              Dates

                Created:
                Updated: