Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-10388

Pure native hadoop client

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • HADOOP-10388
    • None
    • None
    • None

    Description

      A pure native hadoop client has following use case/advantages:
      1. writing Yarn applications using c++
      2. direct access to HDFS, without extra proxy overhead, comparing to web/nfs interface.
      3. wrap native library to support more languages, e.g. python
      4. lightweight, small footprint compare to several hundred MB of JDK and hadoop library with various dependencies.

      Attachments

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. HDFS-6994 libhdfs3 - A native C/C++ HDFS client

          • Major - Major loss of function.
          • Open
          1.
          		 Native RPCv9 client Sub-task Resolved Colin McCabe
          2.
          		 fix hadoop native client CMakeLists.txt issue with older cmakes Sub-task Resolved Wenwu Peng
          3.
          		 Fix some minors error and compile on macosx Sub-task Resolved Binglin Chang
          4.
          		 Add username to native RPCv9 client Sub-task Resolved Colin McCabe
          5.
          		 Add unit test case for net in hadoop native client Sub-task Resolved Wenwu Peng
          6.
          		 Fix some minor typos and add more test cases for hadoop_err Sub-task Resolved Wenwu Peng
          7.
          		 Native Hadoop Client: make clean should remove pb-c.h.s files Sub-task Resolved Binglin Chang
          8.
          		 add pom.xml infrastructure for hadoop-native-core Sub-task Resolved Binglin Chang
          9.
          		 Implement Namenode RPCs in HDFS native client Sub-task Resolved Colin McCabe
          10.
          		 Implement C code for parsing Hadoop / HDFS URIs Sub-task Resolved Colin McCabe
          11.
          		 native code for reading Hadoop configuration XML files Sub-task Resolved Colin McCabe
          12.
          		 Fix initialization of hrpc_sync_ctx Sub-task Resolved Binglin Chang
          13.
          		 limit symbol visibility in libhdfs-core.so and libyarn-core.so Sub-task Resolved Colin McCabe
          14.
          		 Native Hadoop Client:add unit test case for call&client_id Sub-task Resolved Wenwu Peng
          15.
          		 implement TCP connection reuse for native client Sub-task Resolved Colin McCabe
          16.
          		 Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind) Sub-task Resolved Wenwu Peng
          17.
          		 ndfs hdfsDelete should check the return boolean Sub-task Resolved Colin McCabe
          18.
          		 ndfs: need to implement umask, pass permission bits to hdfsCreateDirectory Sub-task Resolved Colin McCabe
          19.
          		 native client: refactor URI code to be clearer Sub-task Resolved Colin McCabe
          20.
          		 Implement listStatus and getFileInfo in the native client Sub-task Resolved Colin McCabe
          21.
          		 native client: implement hdfsMove and hdfsCopy Sub-task Resolved Colin McCabe
          22.
          		 native client: split ndfs.c into meta, file, util, and permission Sub-task Resolved Colin McCabe
          23.
          		 Implement DataTransferProtocol in libhdfs-core.so Sub-task Open Unassigned
          24.
          		 Pure Native Client: implement C code native_mini for YARN for unit test Sub-task Open Wenwu Peng
          25.
          		 native client: parse Hadoop permission strings Sub-task Open Unassigned
          26.
          		 hconf.c: fix bug where we would sometimes not try to load multiple XML files from the same path Sub-task Patch Available Colin McCabe
          27.
          		 implement ndfs_get_hosts Sub-task In Progress Colin McCabe

          Activity

            People

              cmccabe Colin McCabe
              decster Binglin Chang
              Votes:
              1 Vote for this issue
              Watchers:
              44 Start watching this issue

              Dates

                Created:
                Updated: