Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-12991

LLAP: Add interface to allow other applications to retrieve data

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Expose an interface for retrieving data from LLAP. This could be used by applications such as Pig or Spark if they wish to get their data using LLAP.

      Will use the LLAP branch to commit progress.

        Attachments

        1.
        Rebase LLAP branch with master Sub-task Resolved Jason Dere
        2.
        Create initial InputFormat + record readers/writers Sub-task Resolved Gunther Hagleitner
        3.
        Add client to communicate with interface, initial split setup Sub-task Resolved Siddharth Seth
        4.
        Wire the client to submit execution fragments Sub-task Resolved Siddharth Seth
        5.
        Make HiveSplitGenerator usable independent of Tez Sub-task Open Unassigned
        6.
        JDBC split refactoring and handle some edge cases Sub-task Resolved Vikram Dixit K
        7.
        Fixes for LlapDump and FileSinkoperator Sub-task Resolved Gunther Hagleitner
        8.
        Merge master into llap branch Sub-task Resolved Jason Dere
        9.
        LLAP: Add end-to-end test for LlapInputFormat Sub-task Resolved Jason Dere
        10.
        LLAP: Replace use of ServerSocket with netty in LlapOutputFormatService Sub-task Resolved Jason Dere
        11.
        Merge master into llap branch Sub-task Resolved Jason Dere
        12.
        LlapInputFormat should get LLAP ports from the LLAP service registry Sub-task Resolved Jason Dere
        13.
        LlapTaskUmbilicalExternalClient should handle submission rejection/failures/timeouts from LLAP daemon Sub-task Resolved Jason Dere
        14.
        LLAP external submission client ends up attempting to find an LLAP instance based on the submitting user instead of the hive user Sub-task Resolved Jason Dere
        15.
        Merge master into llap branch Sub-task Resolved Jason Dere
        16.
        LLAP output format service not actually registered in LLAP registry Sub-task Resolved Jason Dere
        17.
        Allow LlapRecordReader to parse/output rows Sub-task Resolved Jason Dere
        18.
        Move around some of the classes created during llap branch work Sub-task Resolved Jason Dere
        19.
        Get rid of wrapped LlapInputSplit/InputFormat classes Sub-task Resolved Jason Dere
        20.
        Update LlapDump Sub-task Resolved Jason Dere
        21.
        Misc cleanup on llap branch Sub-task Resolved Jason Dere
        22.
        Merge llap branch work to master Sub-task Closed Jason Dere
        23.
        LLAP: Work out the details of LlapTaskUmbilicalExternalClient tokenIdentifier Sub-task Open Unassigned
        24.
        LLAP: Other todo items in LlapTaskUmbilicalExternalClient Sub-task Open Unassigned
        25.
        LlapBaseInputFormat: figure out where credentials come from Sub-task Open Unassigned
        26.
        Use loginUser from UGI to get llap user when generating LLAP splits. Sub-task Resolved Jason Dere
        27.
        LLAP: handle case where no service instance is found on the host specified in the input split Sub-task Closed Jason Dere
        28.
        usingTezAm field not required in LLAP SubmitWorkRequestProto Sub-task Resolved Jason Dere
        29.
        LlapOutputFormatService port should be able to be set via conf Sub-task Resolved Jason Dere
        30.
        Fix TestLlapOutputFormat Sub-task Resolved Jason Dere
        31.
        Security for LlapOutputFormatService Sub-task Resolved Unassigned
        32.
        LlapTaskUmbilicalExternalClient should be closed by the record reader Sub-task Closed Jason Dere
        33.
        HIVE_RPC_QUERY_PLAN should always be set when generating LLAP splits Sub-task Closed Jason Dere
        34.
        NPE in TaskLocationHints during LLAP GetSplits request Sub-task Closed Jason Dere
        35.
        LLAP: external client output is writing to channel before it is writable again Sub-task Closed Jason Dere
        36.
        LLAP input split should get task attempt number from conf if available Sub-task Closed Jason Dere
        37.
        LLAP output format connection should wait for all writes to finish before closing channel Sub-task Closed Jason Dere
        38.
        LLAP external recordreader not returning non-ascii string properly Sub-task Closed Jason Dere
        39.
        Disable LlapZookeeperRegistry ZK auth setup for external clients Sub-task Closed Jason Dere
        40.
        LLAP external client on secure cluster: Protocol interface org.apache.hadoop.hive.llap.protocol.LlapTaskUmbilicalProtocol is not known Sub-task Closed Jason Dere
        41.
        LLAP input format should escape the query string being passed to getSplits() Sub-task Closed Jason Dere
        42.
        Clean up driver instance in get_splits Sub-task Closed Jason Dere
        43.
        Support complex types in external LLAP InputFormat Sub-task Closed Jason Dere
        44.
        Improve end-of-data checking for LLAP input format Sub-task Closed Jason Dere
        45.
        LlapProtocolClientProxy stack trace when using llap input format Sub-task Closed Jason Dere
        46.
        LlapInputFormat: Seeing "output error" WARN message Sub-task Closed Jason Dere
        47.
        Add test for more datatypes for LlapInputFormat Sub-task Closed Jason Dere
        48.
        Use LazyBinarySerDe for LLAP InputFormat Sub-task Closed Jason Dere
        49.
        LlapTaskUmbilicalExternalClient should not start new umbilical server for every fragment request Sub-task Closed Jason Dere
        50.
        "Timed out getting readerEvents" error from external LLAP client Sub-task Closed Jason Dere
        51.
        External LLAP client: connection to HS2 should be kept open until explicitly closed Sub-task Closed Jason Dere
        52.
        Set current database for external LLAP interface Sub-task Closed Jason Dere
        53.
        External LLAP client: acquire locks for tables queried directly by LLAP Sub-task Closed Jason Dere

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jdere Jason Dere
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: