[IMPALA-8625] Crash in dockerised tests

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Component/s: Backend
• Labels: crash flaky
• Target Version: Impala 3.3.0
• Epic Color: ghx-label-6
• Assignee: Tim Armstrong
• Reporter: Tim Armstrong
• Created: 05/Jun/19 00:13
• Updated: 24/Jun/19 17:37
• Resolved: 24/Jun/19 17:37

Description

https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/439/

I don't have much in the way of diagnostics, just this fragment in the log impalad.3b630ef7faf3.ubuntu.log.INFO.20190604-185929.11.gz :

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f0f139bc6e0, pid=11, tid=0x00007f0ba23b1700
#
# JRE version: OpenJDK Runtime Environment (8.0_212-b03) (build 1.8.0_212-8u212-b03-0ubuntu1.16.04.1-b03)
# Java VM: OpenJDK 64-Bit Server VM (25.212-b03 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  0x00007f0f139bc6e0
#
# Core dump written. Default location: /opt/impala/core or core.11
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid11.log
  74: client_identifier (string) = "query_test/test_insert_parquet.py::T

Tests that were running:

[gw10] PASSED query_test/test_scanners.py::TestScannersAllTableFormats::test_hdfs_scanner_profile[batch_size: 0 | debug_action: -1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: avro/snap/block]
query_test/test_scanners.py::TestScannersAllTableFormats::test_hdfs_scanner_profile[batch_size: 0 | debug_action: -1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
[gw6] PASSED query_test/test_scanners.py::TestScannersAllTableFormats::test_string_escaping[batch_size: 16 | debug_action: HDFS_SCANNER_THREAD_CHECK_SOFT_MEM_LIMIT:FAIL@0.5 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: hbase/none]
query_test/test_scanners.py::TestScannersAllTableFormats::test_string_escaping[batch_size: 16 | debug_action: -1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: seq/def/block]
[gw3] PASSED query_test/test_insert_parquet.py::TestHdfsParquetTableWriter::test_int64_timestamp_logical_type[protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
query_test/test_insert_parquet.py::TestHdfsParquetTableStatsWriter::test_write_statistics_alltypes[protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
[gw6] PASSED query_test/test_scanners.py::TestScannersAllTableFormats::test_string_escaping[batch_size: 16 | debug_action: -1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: seq/def/block]
query_test/test_scanners.py::TestScannersAllTableFormats::test_string_escaping[batch_size: 16 | debug_action: -1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: kudu/none]
[gw13] PASSED query_test/test_join_queries.py::TestTPCHJoinQueries::test_outer_joins[batch_size: 0 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
query_test/test_join_queries.py::TestSemiJoinQueries::test_semi_joins[batch_size: 0 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
[gw6] PASSED query_test/test_scanners.py::TestScannersAllTableFormats::test_string_escaping[batch_size: 16 | debug_action: -1:OPEN:SET_DENY_RESERVATION_PROBABILITY@0.5 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: kudu/none]
query_test/test_scanners.py::TestScannersAllTableFormats::test_string_escaping[batch_size: 16 | debug_action: None | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
[gw6] PASSED query_test/test_scanners.py::TestScannersAllTableFormats::test_string_escaping[batch_size: 16 | debug_action: None | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
query_test/test_scanners.py::TestScannersAllTableFormatsWithLimit::test_limit[protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: text/none]
[gw5] FAILED query_test/test_decimal_fuzz.py::TestDecimalFuzz::test_decimal_ops[exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

]
query_test/test_decimal_fuzz.py::TestDecimalFuzz::test_width_bucket[exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

]
[gw5] FAILED query_test/test_decimal_fuzz.py::TestDecimalFuzz::test_width_bucket[exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

]
query_test/test_decimal_queries.py::TestDecimalQueries::test_queries[protocol: beeswax | exec_option:

{'disable_codegen_rows_threshold': 0, 'disable_codegen': 'false', 'decimal_v2': 'false', 'batch_size': 0}

| table_format: text/none]
[gw11] FAILED query_test/test_queries.py::TestQueries::test_analytic_fns[protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
[gw7] FAILED query_test/test_mem_usage_scaling.py::TestExchangeMemUsage::test_exchange_mem_usage_scaling[protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
query_test/test_mt_dop.py::TestMtDop::test_mt_dop[mt_dop: 0 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: text/none]
[gw1] FAILED query_test/test_scanners.py::TestScannersAllTableFormats::test_scanners[batch_size: 1 | debug_action: HDFS_SCANNER_THREAD_CHECK_SOFT_MEM_LIMIT:FAIL@0.5 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: rc/gzip/block]
query_test/test_scanners.py::TestScannersAllTableFormats::test_scanners[batch_size: 16 | debug_action: HDFS_SCANNER_THREAD_CHECK_SOFT_MEM_LIMIT:FAIL@0.5 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: orc/def/block]
[gw9] FAILED query_test/test_queries.py::TestHdfsQueries::test_file_partitions[protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
query_test/test_queries.py::TestHdfsQueries::test_file_partitions[protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: avro/snap/block]
query_test/test_queries.py::TestQueries::test_analytic_fns[protocol: hs2 | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
[gw5] FAILED query_test/test_decimal_queries.py::TestDecimalQueries::test_queries[protocol: beeswax | exec_option:

{'disable_codegen_rows_threshold': 0, 'disable_codegen': 'false', 'decimal_v2': 'false', 'batch_size': 0}

| table_format: text/none]
query_test/test_decimal_queries.py::TestDecimalQueries::test_queries[protocol: hs2 | exec_option:

{'disable_codegen_rows_threshold': 0, 'disable_codegen': 'true', 'decimal_v2': 'false', 'batch_size': 0}

| table_format: parquet/none]
[gw15] FAILED query_test/test_mt_dop.py::TestMtDopParquet::test_parquet_filtering[mt_dop: 8 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
query_test/test_mt_dop.py::TestMtDopKudu::test_kudu[mt_dop: 0 | protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: text/none]
[gw0] FAILED query_test/test_nested_types.py::TestNestedTypes::test_tpch[protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: parquet/none]
query_test/test_nested_types.py::TestNestedTypes::test_tpch_limit[protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: orc/def/block]
[gw6] FAILED query_test/test_scanners.py::TestScannersAllTableFormatsWithLimit::test_limit[protocol: beeswax | exec_option:

{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0}

| table_format: text/none]