[IMPALA-6640] profile collection via standard Impyla usage won't have ExecSummary

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 2.11.0
• Component/s: Infrastructure
• Labels: profile
• Target Version: Product Backlog
• Epic Color: ghx-label-8
• Assignee: Unassigned
• Reporter: Michael Brown
• Created: 12/Mar/18 17:21
• Updated: 14/May/20 19:16
• Resolved: 14/May/20 19:16

Description

It seems the Impyla (HS2 client for Python stress test) DB API paradigm doesn't lend itself to the ideal collection of query profiles.

1. It seems a profile will not have an ExecSummary, End Time, or Errors unless the profile is requested after a CloseOperation RPC. This makes sense to me.
2. GetRuntimeProfile RPC must be called before the CloseSession RPC. This makes sense to me.
3. One can't really use standard, public methods in Impyla to do this: impala.hiveserver2.HiveServer2Cursor().close() handles both the CloseOperation and CloseSession RPCs in a single call.
4. The same is true of the cursor in a context manager _exit_ setting. _exit_ just calls close()

This means we have to tap into the more private bits of Impyla to do this.

#!/usr/bin/env impala-python

import impala.dbapi
from impala._thrift_api import TGetRuntimeProfileReq

with impala.dbapi.connect(host='localhost', port=21050) as conn:
  with conn.cursor() as cursor:
    cursor.execute('SELECT COUNT(2) FROM functional.alltypes')
    cursor.fetchall()
    with open('profile-before-close-operation.txt', 'w') as fh:
      fh.write(cursor.get_profile())
    # Normal cursor context would end here, but....
    # Issue CloseOperation RPC without CloseSession RPC.
    # This is impala.hiveserver2.Operation().close()
    resp = cursor._last_operation.close()
    req = TGetRuntimeProfileReq(operationHandle=cursor._last_operation.handle,
                                sessionHandle=cursor._last_operation.session.handle)
    resp = cursor._last_operation._rpc('GetRuntimeProfile', req)
    # impala.hiveserver2.HiveServer2Cursor()._reset_state() assist
    cursor._last_operation_active = False
    with open('profile-after-close-operation.txt', 'w') as fh:
      fh.write(resp.profile)

We should think about pythonic ways to make this better for Impyla, or at least alter the stress test to get the profiles of the queries after CloseOperation.

Issue Links

• duplicates: IMPALA-6742 Profiles of running queries should include execution summary