Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2234

Exceeding mem limit may result in "Cancelled" failure

    Details

      Description

      The "Cancelled" message should instead say something about a mem limit being exceeded. The "Cancelled" message is a bit confusing since the user did not cancel the query.

      Coordinator log

       grep '554f6c3731c1d173:d20b4d8b807fff9a' /var/log/impalad/impalad.INFO 
      I0821 17:15:19.122673  5618 coordinator.cc:309] Exec() query_id=554f6c3731c1d173:d20b4d8b807fff9a
      I0821 17:15:19.212877  5618 plan-fragment-executor.cc:82] Prepare(): query_id=554f6c3731c1d173:d20b4d8b807fff9a instance_id=554f6c3731c1d173:d20b4d8b807fff9b
      I0821 17:15:19.379251  5618 coordinator.cc:385] starting 18 backends for query 554f6c3731c1d173:d20b4d8b807fff9a
      I0821 17:15:19.871928 22621 plan-fragment-executor.cc:82] Prepare(): query_id=554f6c3731c1d173:d20b4d8b807fff9a instance_id=554f6c3731c1d173:d20b4d8b807fff9d
      I0821 17:15:30.211174 17447 plan-fragment-executor.cc:82] Prepare(): query_id=554f6c3731c1d173:d20b4d8b807fff9a instance_id=554f6c3731c1d173:d20b4d8b807fffa1
      I0821 17:15:30.846810 22553 plan-fragment-executor.cc:82] Prepare(): query_id=554f6c3731c1d173:d20b4d8b807fff9a instance_id=554f6c3731c1d173:d20b4d8b807fffa3
      I0821 17:15:41.101415 10432 plan-fragment-executor.cc:82] Prepare(): query_id=554f6c3731c1d173:d20b4d8b807fff9a instance_id=554f6c3731c1d173:d20b4d8b807fffa7
      I0821 17:15:46.802021  5487 buffered-block-mgr.cc:769] Query: 554f6c3731c1d173:d20b4d8b807fff9a write complete callback with error.
      I0821 17:15:46.802042  5487 runtime-state.cc:230] Error from query 554f6c3731c1d173:d20b4d8b807fff9a: Memory limit exceeded
      I0821 17:15:47.026504  8744 runtime-state.cc:230] Error from query 554f6c3731c1d173:d20b4d8b807fff9a: Memory Limit Exceeded
      Query(554f6c3731c1d173:d20b4d8b807fff9a) Limit: Consumption=296.41 MB
      I0821 17:15:49.310215  8502 runtime-state.cc:230] Error from query 554f6c3731c1d173:d20b4d8b807fff9a: Memory Limit Exceeded
      I0821 17:15:52.537783  5618 coordinator.cc:1116] Cancel() query_id=554f6c3731c1d173:d20b4d8b807fff9a
      I0821 17:15:52.549916 22570 coordinator.cc:1301] Backend 4 completed, 17 remaining: query_id=554f6c3731c1d173:d20b4d8b807fff9a
      I0821 17:15:52.549937 22570 coordinator.cc:1310] query_id=554f6c3731c1d173:d20b4d8b807fff9a: first in-progress backend: impala-stress-4.vpc.cloudera.com:22000
      Query(554f6c3731c1d173:d20b4d8b807fff9a) Limit: Consumption=296.41 MB
      I0821 17:15:52.557632  5618 impala-server.cc:813] UnregisterQuery(): query_id=554f6c3731c1d173:d20b4d8b807fff9a
      I0821 17:15:52.557718  5618 impala-server.cc:882] Cancel(): query_id=554f6c3731c1d173:d20b4d8b807fff9a
      I0821 17:15:52.565330 17443 impala-server.cc:985] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled). (query_id: 554f6c3731c1d173:d20b4d8b807fff9a, backend: 7, instance: 554f6c3731c1d173:d20b4d8b807fffa3 done: true)
      I0821 17:15:52.947180 22548 impala-server.cc:985] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled). (query_id: 554f6c3731c1d173:d20b4d8b807fff9a, backend: 13, instance: 554f6c3731c1d173:d20b4d8b807fffa9 done: true)
      I0821 17:15:52.947417 24348 impala-server.cc:985] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled). (query_id: 554f6c3731c1d173:d20b4d8b807fff9a, backend: 2, instance: 554f6c3731c1d173:d20b4d8b807fff9e done: true)
      I0821 17:15:53.522819  5866 impala-server.cc:985] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled). (query_id: 554f6c3731c1d173:d20b4d8b807fff9a, backend: 10, instance: 554f6c3731c1d173:d20b4d8b807fffa6 done: true)
      I0821 17:16:00.691387  6208 impala-server.cc:985] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled). (query_id: 554f6c3731c1d173:d20b4d8b807fff9a, backend: 12, instance: 554f6c3731c1d173:d20b4d8b807fffa8 done: false)
      I0821 17:16:01.623899  5907 impala-server.cc:985] ReportExecStatus(): Received report for unknown query ID (probably closed or cancelled). (query_id: 554f6c3731c1d173:d20b4d8b807fff9a, backend: 12, instance: 554f6c3731c1d173:d20b4d8b807fffa8 done: true)
      

      Profile

      Query (id=554f6c3731c1d173:d20b4d8b807fff9a):
        Summary:
          Session ID: a146dd618550b188:8b1ba469daa09789
          Session Type: HIVESERVER2
          HiveServer2 Protocol Version: V6
          Start Time: 2015-08-21 17:15:18.180530000
          End Time: 2015-08-21 17:15:52.558928000
          Query Type: QUERY
          Query State: EXCEPTION
          Query Status: 
      Cancelled
      
          Impala Version: impalad version 2.2.0-cdh5 DEBUG (build 2ffd73a4255cefd521362ffe1cfb37463f67f75c)
      ....
      

      I'm sure this query was not cancelled by a client rpc.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                caseyc casey
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: