Uploaded image for project: 'Apache Lens (Retired)'
  1. Apache Lens (Retired)
  2. LENS-1165

Separate client for query submission from other thrift operations

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: driver-hive
    • Labels:
      None

      Description

      Proposing to add separate thrift client for query submissions in HiveDriver with ability to give different thrift read timeouts and number of retries.

      This would help in the following cases :
      We have seen instances of query submission which can take more time. For ex :


      2016-06-02T20:20:21,436 INFO [ef431743-b9f8-4fe0-8e3d-2bab154438eb HiveServer2-Handler-Pool: Thread-66837]: ql.Driver (Driver.java:compile(372)) - Compiling command(queryId=hive_20160602195701_2a3bd795-0919-4e2e-ad22-812881b084fd):
      2016-06-02T21:02:43,380 INFO [ef431743-b9f8-4fe0-8e3d-2bab154438eb HiveServer2-Handler-Pool: Thread-66255]: ql.Driver (Driver.java:compile(524)) - Completed compiling command(queryId=hive_20160602200002_e58e07f9-61d0-423d-86ef-a626eb250114); Time taken: 1830.214 seconds

      In such cases lens submission will timeout with default client and resubmit, which would take more time again.
      And eventually, there will be number of submissions equal to the number of retries configured and lens would have marked the query as failed. But submission would actually launch multiple queries.

      If we have separate client, it can be configured with different timeout value and number retries to be different.

        Attachments

          Activity

            People

            • Assignee:
              amareshwari Amareshwari Sriramadasu
              Reporter:
              amareshwari Amareshwari Sriramadasu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: