Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: HADOOP-10388
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      A pure native hadoop client has following use case/advantages:
      1. writing Yarn applications using c++
      2. direct access to HDFS, without extra proxy overhead, comparing to web/nfs interface.
      3. wrap native library to support more languages, e.g. python
      4. lightweight, small footprint compare to several hundred MB of JDK and hadoop library with various dependencies.

        Issue Links

        1.
        Native RPCv9 client Sub-task Resolved Colin Patrick McCabe
         
        2.
        fix hadoop native client CMakeLists.txt issue with older cmakes Sub-task Resolved Wenwu Peng
         
        3.
        Fix some minors error and compile on macosx Sub-task Resolved Binglin Chang
         
        4.
        Add username to native RPCv9 client Sub-task Resolved Colin Patrick McCabe
         
        5.
        Add unit test case for net in hadoop native client Sub-task Resolved Wenwu Peng
         
        6.
        Fix some minor typos and add more test cases for hadoop_err Sub-task Resolved Wenwu Peng
         
        7.
        Native Hadoop Client: make clean should remove pb-c.h.s files Sub-task Resolved Binglin Chang
         
        8.
        add pom.xml infrastructure for hadoop-native-core Sub-task Resolved Binglin Chang
         
        9.
        Implement Namenode RPCs in HDFS native client Sub-task Resolved Colin Patrick McCabe
         
        10.
        Implement C code for parsing Hadoop / HDFS URIs Sub-task Resolved Colin Patrick McCabe
         
        11.
        native code for reading Hadoop configuration XML files Sub-task Resolved Colin Patrick McCabe
         
        12.
        Fix initialization of hrpc_sync_ctx Sub-task Resolved Binglin Chang
         
        13.
        limit symbol visibility in libhdfs-core.so and libyarn-core.so Sub-task Resolved Colin Patrick McCabe
         
        14.
        Native Hadoop Client:add unit test case for call&client_id Sub-task Resolved Wenwu Peng
         
        15.
        implement TCP connection reuse for native client Sub-task Resolved Colin Patrick McCabe
         
        16.
        Fix namenode-rpc-unit warning reported by memory leak check tool(valgrind) Sub-task Resolved Wenwu Peng
         
        17.
        ndfs hdfsDelete should check the return boolean Sub-task Resolved Colin Patrick McCabe
         
        18.
        ndfs: need to implement umask, pass permission bits to hdfsCreateDirectory Sub-task Resolved Colin Patrick McCabe
         
        19.
        native client: refactor URI code to be clearer Sub-task Resolved Colin Patrick McCabe
         
        20.
        Implement listStatus and getFileInfo in the native client Sub-task Resolved Colin Patrick McCabe
         
        21.
        native client: implement hdfsMove and hdfsCopy Sub-task Resolved Colin Patrick McCabe
         
        22.
        native client: split ndfs.c into meta, file, util, and permission Sub-task Resolved Colin Patrick McCabe
         
        23. Implement DataTransferProtocol in libhdfs-core.so Sub-task Open Unassigned
         
        24. Pure Native Client: implement C code native_mini for YARN for unit test Sub-task Open Wenwu Peng
         
        25. native client: parse Hadoop permission strings Sub-task Open Unassigned
         
        26. hconf.c: fix bug where we would sometimes not try to load multiple XML files from the same path Sub-task Patch Available Colin Patrick McCabe
         
        27. implement ndfs_get_hosts Sub-task In Progress Colin Patrick McCabe
         

          Activity

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Binglin Chang
            • Votes:
              0 Vote for this issue
              Watchers:
              42 Start watching this issue

              Dates

              • Created:
                Updated:

                Development