Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7983

Failed verification/decryption while reading spilled data causing test failure

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Impala 3.1.0
    • Fix Version/s: None
    • Component/s: Backend
    • Labels:

      Description

      Failed test:

      query_test.test_spilling.TestSpillingDebugActionDimensions.test_spilling[protocol: beeswax | exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0', 'default_spillable_buffer_size': '256k'} | table_format: parquet/none]
      

      Build type: exhaustive/centos6

      Build logs:

      query_test/test_spilling.py:59: in test_spilling     self.run_test_case('QueryTest/spilling', vector) common/impala_test_suite.py:467: in run_test_case     result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:688: in __execute_query     return impalad_client.execute(query, user=user) common/impala_connection.py:170: in execute     return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:182: in execute     handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:359: in __execute_query     self.wait_for_finished(handle) beeswax/impala_beeswax.py:380: in wait_for_finished     raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E   ImpalaBeeswaxException: ImpalaBeeswaxException: E    Query aborted:Error reading 524288 bytes from scratch file '/tmp/impala-scratch/4b417a5b42d94b12:2ba2321000000000_d1ff959f-84da-4176-8b29-ff8e1560e941' on backend shared-centos64-ec2-m2-4xlarge-ondemand-1d0a.vpc.cloudera.com:22001 at offset 806092800: verification of read data failed. E   OpenSSL error in EVP_DecryptFinal decrypting:
      

      Stacktrace

      query_test/test_spilling.py:59: in test_spilling
          self.run_test_case('QueryTest/spilling', vector)
      common/impala_test_suite.py:467: in run_test_case
          result = self.__execute_query(target_impalad_client, query, user=user)
      common/impala_test_suite.py:688: in __execute_query
          return impalad_client.execute(query, user=user)
      common/impala_connection.py:170: in execute
          return self.__beeswax_client.execute(sql_stmt, user=user)
      beeswax/impala_beeswax.py:182: in execute
          handle = self.__execute_query(query_string.strip(), user=user)
      beeswax/impala_beeswax.py:359: in __execute_query
          self.wait_for_finished(handle)
      beeswax/impala_beeswax.py:380: in wait_for_finished
          raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
      E   ImpalaBeeswaxException: ImpalaBeeswaxException:
      E    Query aborted:Error reading 524288 bytes from scratch file '/tmp/impala-scratch/4b417a5b42d94b12:2ba2321000000000_d1ff959f-84da-4176-8b29-ff8e1560e941' on backend shared-centos64-ec2-m2-4xlarge-ondemand-1d0a.vpc.cloudera.com:22001 at offset 806092800: verification of read data failed.
      E   OpenSSL error in EVP_DecryptFinal decrypting:
      

      Build stderr:

      -- executing against localhost:21000
      use tpch_parquet;
      
      -- 2018-12-14 00:17:21,945 INFO     MainThread: Started query 244b8ebf1a985cca:df5b0e3a00000000
      SET debug_action=-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0;
      SET default_spillable_buffer_size=256k;
      -- executing against localhost:21000
      set buffer_pool_limit=215m;
      
      -- 2018-12-14 00:17:21,990 INFO     MainThread: Started query 1a46eec9e244c88a:49a6b2df00000000
      -- executing against localhost:21000
      
      select count(l1.l_tax)
      from
      lineitem l1,
      lineitem l2,
      lineitem l3
      where
      l1.l_tax < 0.01 and
      l2.l_tax < 0.04 and
      l1.l_orderkey = l2.l_orderkey and
      l1.l_orderkey = l3.l_orderkey and
      l1.l_comment = l3.l_comment and
      l1.l_shipdate = l3.l_shipdate;
      
      -- 2018-12-14 00:17:22,059 INFO     MainThread: Started query ef4546e5750f98b3:d67f771600000000
      -- executing against localhost:21000
      SET BUFFER_POOL_LIMIT="";
      
      -- 2018-12-14 00:17:47,105 INFO     MainThread: Started query b84dc37674cb5ab9:e080595b00000000
      -- executing against localhost:21000
      set buffer_pool_limit=50m;
      
      -- 2018-12-14 00:17:47,132 INFO     MainThread: Started query be4b201e2b52176a:2f88f48300000000
      -- executing against localhost:21000
      
      select max(t1.total_count), max(t1.l_shipinstruct), max(t1.l_comment) from
      (select l_shipinstruct, l_comment, count(*) over () total_count from lineitem) t1;
      
      -- 2018-12-14 00:17:47,213 INFO     MainThread: Started query 1d4f0a32074b6174:5f21f35700000000
      -- executing against localhost:21000
      SET BUFFER_POOL_LIMIT="";
      
      -- 2018-12-14 00:18:11,983 INFO     MainThread: Started query fa4364af8ab07926:b2cde42000000000
      -- executing against localhost:21000
      set buffer_pool_limit=20m;
      
      -- 2018-12-14 00:18:11,997 INFO     MainThread: Started query 1143a7d66b2b11df:c04d2fdf00000000
      -- executing against localhost:21000
      
      select a.int_col, count(*)
      from functional.alltypessmall a, functional.alltypessmall b, functional.alltypessmall c
      where a.id = b.id and b.id = c.id group by a.int_col;
      
      -- 2018-12-14 00:18:12,015 INFO     MainThread: Started query c94b5a6145d5b924:60b5939100000000
      -- executing against localhost:21000
      SET BUFFER_POOL_LIMIT="";
      
      -- 2018-12-14 00:18:12,340 INFO     MainThread: Started query c74fe6cefcd9dd86:64e37ce100000000
      -- 2018-12-14 00:18:12,348 INFO     MainThread: Query Name: 
      TPCH-Q21
      
      -- executing against localhost:21000
      set buffer_pool_limit=100m;
      
      -- 2018-12-14 00:18:12,356 INFO     MainThread: Started query 5f4300a672e4b48f:6c6477ea00000000
      -- executing against localhost:21000
      
      select
        s_name,
        count(*) as numwait
      from
        supplier,
        lineitem l1 join [BROADCAST]
        orders,
        nation
      where
        s_suppkey = l1.l_suppkey
        and o_orderkey = l1.l_orderkey
        and o_orderstatus = 'F'
        and l1.l_receiptdate > l1.l_commitdate
        and exists (
          select
            *
          from
            lineitem l2
          where
            l2.l_orderkey = l1.l_orderkey
            and l2.l_suppkey <> l1.l_suppkey
        )
        and not exists (
          select
            *
          from
            lineitem l3
          where
            l3.l_orderkey = l1.l_orderkey
            and l3.l_suppkey <> l1.l_suppkey
            and l3.l_receiptdate > l3.l_commitdate
        )
        and s_nationkey = n_nationkey
        and n_name = 'SAUDI ARABIA'
      group by
        s_name
      order by
        numwait desc,
        s_name
      limit 100;
      
      -- 2018-12-14 00:18:12,452 INFO     MainThread: Started query 134aa693a8e4fed5:e4e21c7000000000
      -- executing against localhost:21000
      SET BUFFER_POOL_LIMIT="";
      
      -- 2018-12-14 00:18:47,992 INFO     MainThread: Started query 3c419c8cc9242655:956aae7500000000
      -- executing against localhost:21000
      set buffer_pool_limit=170m;
      
      -- 2018-12-14 00:18:48,104 INFO     MainThread: Started query 8646a7bfbe58b96f:1acdb73000000000
      -- executing against localhost:21000
      
      set disable_outermost_topn=1;
      
      -- 2018-12-14 00:18:48,132 INFO     MainThread: Started query f14f5ee0e6988761:7f5815c200000000
      -- executing against localhost:21000
      
      select * from lineitem
        inner join orders on l_orderkey = o_orderkey
      order by l_linenumber, l_suppkey, l_partkey, l_orderkey
      limit 20;
      
      -- 2018-12-14 00:18:48,203 INFO     MainThread: Started query 4b417a5b42d94b12:2ba2321000000000
      -- executing against localhost:21000
      SET BUFFER_POOL_LIMIT="";
      
      -- 2018-12-14 00:25:37,632 INFO     MainThread: Started query 9d4396efceba3f14:df6e2b6200000000
      -- executing against localhost:21000
      SET DISABLE_OUTERMOST_TOPN="0";
      
      -- 2018-12-14 00:25:37,707 INFO     MainThread: Started query 1244360bf6a6b531:bfb5d65b00000000
      

      Couldn't find a smoking gun in the logs.

      I1214 00:25:37.534629 31548 status.cc:128] OpenSSL error in EVP_DecryptFinal decrypting:
          @          0x1a36b7a
          @          0x21843bd
          @          0x2184f15
          @          0x2184697
          @          0x1ef7f95
          @          0x1ef617a
          @          0x21f4162
          @          0x21f071b
          @          0x272446f
          @          0x271bbc9
          @          0x272916f
          @          0x271b4c9
          @          0x27308f3
          @          0x2730587
          @          0x272fe48
          @          0x272f51b
          @          0x1ee06d7
          @          0x1ee026b
          @          0x1edfb7b
          @          0x1edf780
          @          0x2723378
          @          0x2722b49
          @          0x27219c9
          @          0x2371c0d
          @          0x2370853
          @          0x1f1e53f
          @          0x1f1b773
          @          0x1f2c148
          @          0x1f2a451
          @          0x1f2cef0
          @          0x1d44705
          @          0x21ca1c2
      I1214 00:25:37.534849 31548 status.cc:77] Error reading 524288 bytes from scratch file '/tmp/impala-scratch/4b417a5b42d94b12:2ba2321000000000_d1ff959f-84da-4176-8b29-ff8e1560e941' on backend shared-centos64-ec2-m2-4xlarge-ondemand-1d0a.vpc.cloudera.com:22001 at offset 806092800: verification of read data failed.
          @          0x1a35d7b
          @          0x1ef8089
          @          0x1ef617a
          @          0x21f4162
          @          0x21f071b
          @          0x272446f
          @          0x271bbc9
          @          0x272916f
          @          0x271b4c9
          @          0x27308f3
          @          0x2730587
          @          0x272fe48
          @          0x272f51b
          @          0x1ee06d7
          @          0x1ee026b
          @          0x1edfb7b
          @          0x1edf780
          @          0x2723378
          @          0x2722b49
          @          0x27219c9
          @          0x2371c0d
          @          0x2370853
          @          0x1f1e53f
          @          0x1f1b773
          @          0x1f2c148
          @          0x1f2a451
          @          0x1f2cef0
          @          0x1d44705
          @          0x21ca1c2
          @          0x21d24e2
          @          0x21d2406
          @          0x21d23c9
      I1214 00:25:37.554332 31545 query-state.cc:249] UpdateBackendExecState(): last report for 4b417a5b42d94b12:2ba2321000000000
      I1214 00:25:37.558511 31548 query-state.cc:576] Instance completed. instance_id=4b417a5b42d94b12:2ba2321000000005 #in-flight=3 status=SCRATCH_READ_VERIFY_FAILED: Error reading 524288 bytes from scratch file '/tmp/impala-scratch/4b417a5b42d94b12:2ba2321000000000_d1ff959f-84da-4176-8b29-ff8e1560e941' on backend shared-centos64-ec2-m2-4xlarge-ondemand-1d0a.vpc.cloudera.com:22001 at offset 806092800: verification of read data failed.
      OpenSSL error in EVP_DecryptFinal decrypting:
      I1214 00:25:37.568645 31545 query-state.cc:383] Cancelling fragment instances as directed by the coordinator. Returned status: Cancelled
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              bharathv Bharath Vissapragada
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: