Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12991

LLAP: Add interface to allow other applications to retrieve data

Log workAgile BoardRank to TopRank to BottomAdd voteVotersWatch issueWatchersCreate sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Expose an interface for retrieving data from LLAP. This could be used by applications such as Pig or Spark if they wish to get their data using LLAP.

      Will use the LLAP branch to commit progress.

        Attachments

        1.
        Rebase LLAP branch with master Sub-task Resolved Jason Dere Actions
        2.
        Create initial InputFormat + record readers/writers Sub-task Resolved Gunther Hagleitner Actions
        3.
        Add client to communicate with interface, initial split setup Sub-task Resolved Siddharth Seth Actions
        4.
        Wire the client to submit execution fragments Sub-task Resolved Siddharth Seth Actions
        5.
        Make HiveSplitGenerator usable independent of Tez Sub-task Open Unassigned Actions
        6.
        JDBC split refactoring and handle some edge cases Sub-task Resolved Vikram Dixit K Actions
        7.
        Fixes for LlapDump and FileSinkoperator Sub-task Resolved Gunther Hagleitner Actions
        8.
        Merge master into llap branch Sub-task Resolved Jason Dere Actions
        9.
        LLAP: Add end-to-end test for LlapInputFormat Sub-task Resolved Jason Dere Actions
        10.
        LLAP: Replace use of ServerSocket with netty in LlapOutputFormatService Sub-task Resolved Jason Dere Actions
        11.
        Merge master into llap branch Sub-task Resolved Jason Dere Actions
        12.
        LlapInputFormat should get LLAP ports from the LLAP service registry Sub-task Resolved Jason Dere Actions
        13.
        LlapTaskUmbilicalExternalClient should handle submission rejection/failures/timeouts from LLAP daemon Sub-task Resolved Jason Dere Actions
        14.
        LLAP external submission client ends up attempting to find an LLAP instance based on the submitting user instead of the hive user Sub-task Resolved Jason Dere Actions
        15.
        Merge master into llap branch Sub-task Resolved Jason Dere Actions
        16.
        LLAP output format service not actually registered in LLAP registry Sub-task Resolved Jason Dere Actions
        17.
        Allow LlapRecordReader to parse/output rows Sub-task Resolved Jason Dere Actions
        18.
        Move around some of the classes created during llap branch work Sub-task Resolved Jason Dere Actions
        19.
        Get rid of wrapped LlapInputSplit/InputFormat classes Sub-task Resolved Jason Dere Actions
        20.
        Update LlapDump Sub-task Resolved Jason Dere Actions
        21.
        Misc cleanup on llap branch Sub-task Resolved Jason Dere Actions
        22.
        Merge llap branch work to master Sub-task Closed Jason Dere Actions
        23.
        LLAP: Work out the details of LlapTaskUmbilicalExternalClient tokenIdentifier Sub-task Open Unassigned Actions
        24.
        LLAP: Other todo items in LlapTaskUmbilicalExternalClient Sub-task Open Unassigned Actions
        25.
        LlapBaseInputFormat: figure out where credentials come from Sub-task Open Unassigned Actions
        26.
        Use loginUser from UGI to get llap user when generating LLAP splits. Sub-task Resolved Jason Dere Actions
        27.
        LLAP: handle case where no service instance is found on the host specified in the input split Sub-task Closed Jason Dere Actions
        28.
        usingTezAm field not required in LLAP SubmitWorkRequestProto Sub-task Resolved Jason Dere Actions
        29.
        LlapOutputFormatService port should be able to be set via conf Sub-task Resolved Jason Dere Actions
        30.
        Fix TestLlapOutputFormat Sub-task Resolved Jason Dere Actions
        31.
        Security for LlapOutputFormatService Sub-task Resolved Unassigned Actions
        32.
        LlapTaskUmbilicalExternalClient should be closed by the record reader Sub-task Closed Jason Dere Actions
        33.
        HIVE_RPC_QUERY_PLAN should always be set when generating LLAP splits Sub-task Closed Jason Dere Actions
        34.
        NPE in TaskLocationHints during LLAP GetSplits request Sub-task Closed Jason Dere Actions
        35.
        LLAP: external client output is writing to channel before it is writable again Sub-task Closed Jason Dere Actions
        36.
        LLAP input split should get task attempt number from conf if available Sub-task Closed Jason Dere Actions
        37.
        LLAP output format connection should wait for all writes to finish before closing channel Sub-task Closed Jason Dere Actions
        38.
        LLAP external recordreader not returning non-ascii string properly Sub-task Closed Jason Dere Actions
        39.
        Disable LlapZookeeperRegistry ZK auth setup for external clients Sub-task Closed Jason Dere Actions
        40.
        LLAP external client on secure cluster: Protocol interface org.apache.hadoop.hive.llap.protocol.LlapTaskUmbilicalProtocol is not known Sub-task Closed Jason Dere Actions
        41.
        LLAP input format should escape the query string being passed to getSplits() Sub-task Closed Jason Dere Actions
        42.
        Clean up driver instance in get_splits Sub-task Closed Jason Dere Actions
        43.
        Support complex types in external LLAP InputFormat Sub-task Closed Jason Dere Actions
        44.
        Improve end-of-data checking for LLAP input format Sub-task Closed Jason Dere Actions
        45.
        LlapProtocolClientProxy stack trace when using llap input format Sub-task Closed Jason Dere Actions
        46.
        LlapInputFormat: Seeing "output error" WARN message Sub-task Closed Jason Dere Actions
        47.
        Add test for more datatypes for LlapInputFormat Sub-task Closed Jason Dere Actions
        48.
        Use LazyBinarySerDe for LLAP InputFormat Sub-task Closed Jason Dere Actions
        49.
        LlapTaskUmbilicalExternalClient should not start new umbilical server for every fragment request Sub-task Closed Jason Dere Actions
        50.
        "Timed out getting readerEvents" error from external LLAP client Sub-task Closed Jason Dere Actions
        51.
        External LLAP client: connection to HS2 should be kept open until explicitly closed Sub-task Closed Jason Dere Actions
        52.
        Set current database for external LLAP interface Sub-task Closed Jason Dere Actions
        53.
        External LLAP client: acquire locks for tables queried directly by LLAP Sub-task Closed Jason Dere Actions

          Activity

          $i18n.getText('security.level.explanation', $currentSelection) Viewable by All Users
          Cancel

            People

              Dates

              • Created:
                Updated:

                Issue deployment