Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11671

Asynchronous native RPC v9 client

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None

    Description

      There are more and more integration happening between Hadoop and applications that are implemented using languages other than Java.

      To access Hadoop, applications either have to go through JNI (e.g. libhdfs), or to reverse engineer the Hadoop RPC protocol. (e.g. snakebite). Unfortunately, neither of them are satisfactory:

      • Integrating with JNI requires running a JVM inside the application. Some applications (e.g., real-time processing, MPP database) does not want the footprints and GC behavior of the JVM.
      • The Hadoop RPC protocol has a rich feature set including wire encryption, SASL, Kerberos authentication. Many 3rd-party implementations can fully cover the feature sets thus they might work in limited environment.

      This jira is to propose implementing an Hadoop RPC library in C++ that provides a common ground to implement higher-level native client for HDFS, YARN, and MapReduce.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            wheat9 Haohui Mai
            wheat9 Haohui Mai

            Dates

              Created:
              Updated:

              Slack

                Issue deployment