Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3186

Stress test crash: wait until impalad is quiesced before starting next test: [libstdc++.so.6+0x5cad0] __dynamic_cast+0x50

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Impala 2.6.0
    • Fix Version/s: None
    • Component/s: Infrastructure
    • Labels:

      Description

      There was a crash on the EC2 stress test cluster. In the hs_err_pid file we find:

      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      #  SIGSEGV (0xb) at pc=0x00007f61f241dad0, pid=18718, tid=140039557764864
      #
      # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # C  [libstdc++.so.6+0x5cad0]  __dynamic_cast+0x50
      #
      # Core dump written. Default location: /var/log/impalad/core or core.18718
      #
      # If you would like to submit a bug report, please visit:
      #   http://bugreport.sun.com/bugreport/crash.jsp
      

      and in the impalad.ERROR log:

      Log file created at: 2016/03/15 00:04:31
      Running on machine: impala-stress-cdh5-trunk2-2.vpc.cloudera.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      E0315 00:04:31.310118 18718 logging.cc:120] stderr will be logged to this file.
      E0315 07:50:01.608139  6880 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:01.747846  6884 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:02.445477  8224 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:10.240705  8572 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:10.397904  7070 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:12.076678  7198 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:19.973160  8696 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:20.319783  8233 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:29.652158  7819 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:30.150673  7084 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:35.144541  9215 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:35.868957  8489 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-2.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      E0315 07:50:39.297054  7494 data-stream-sender.cc:259] channel send status: Couldn't open transport for impala-stress-cdh5-trunk2-3.vpc.cloudera.com:22000 (connect() failed: Connection timed out)
      Picked up JAVA_TOOL_OPTIONS:
      hdfsOpenFile(hdfs://impala-stress-cdh5-trunk2-1.vpc.cloudera.com:8020/user/hive/warehouse/tpch_3_nested_parquet.db/customer/000001_0): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error:
      FSDataInputStream#close error:
      terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >'
        what():  boost: mutex lock failed in pthread_mutex_lock: Invalid argument
      ~                                                                                          
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: