Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10592

Exhaustive tests timeout after 20 hours

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 4.0.0
    • Fix Version/s: Impala 4.0.0
    • Component/s: None
    • Labels:
      None
    • Epic Color:
      ghx-label-9

      Description

      The tests seem to make progress for nearly 10 hours, but after 20 hours they timeout

      **** run-all-tests.sh TIMED OUT! ****
      

      The timeout stack traces are attached

      Impala logs show a long period of inactivity between 03/16 16:58 and 03/17 04:53
      For example:

      I0316 16:56:33.555305  9911 impala-server.cc:1996] Catalog topic update applied with version: 65701 new min catalog object version: 36078
      I0316 16:58:00.504211  9041 krpc-data-stream-mgr.cc:427] Reduced stream ID cache from 6 items, to 5, eviction took: 0
      I0316 16:58:10.504297  9041 krpc-data-stream-mgr.cc:427] Reduced stream ID cache from 5 items, to 4, eviction took: 0
      I0316 16:58:20.504348  9041 krpc-data-stream-mgr.cc:427] Reduced stream ID cache from 4 items, to 3, eviction took: 0
      I0316 16:58:30.504386  9041 krpc-data-stream-mgr.cc:427] Reduced stream ID cache from 3 items, to 2, eviction took: 0
      I0316 16:58:40.504467  9041 krpc-data-stream-mgr.cc:427] Reduced stream ID cache from 2 items, to 1, eviction took: 0
      I0316 16:58:50.504545  9041 krpc-data-stream-mgr.cc:427] Reduced stream ID cache from 1 items, to 0, eviction took: 0
      I0317 04:53:06.368000  9905 TAcceptQueueServer.cpp:340] New connection to server StatestoreSubscriber from client <Host: ::ffff:127.0.0.1 Port: 32818>
      I0317 04:53:06.368041  9910 thrift-util.cc:96] TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 36850>Broken pipe
      W0317 04:53:06.369092  8780 init.cc:214] A process pause was detected for approximately 18s920ms
      I0317 04:53:06.369904  9905 TAcceptQueueServer.cpp:340] New connection to server StatestoreSubscriber from client <Host: ::ffff:127.0.0.1 Port: 32822>
      I0317 04:53:06.369961  9910 thrift-util.cc:96] TAcceptQueueServer client died: write() send(): Broken pipe
      W0317 04:53:06.369966  8929 JvmPauseMonitor.java:205] Detected pause in JVM or host machine (eg GC): pause of approximately 18338ms
      No GCs detected
      I0317 04:53:06.370081 27248 thrift-util.cc:96] TSocket::write_partial() send() <Host: ::ffff:127.0.0.1 Port: 32818>Broken pipe
      I0317 04:53:06.370126 27248 thrift-util.cc:96] TAcceptQueueServer client died: write() send(): Broken pipe
      

        Attachments

        1. impalad_8744_jstack_20210317-045312.txt
          83 kB
          Andrew Sherman
        2. catalogd_8661_20210317-045247.txt
          25 kB
          Andrew Sherman
        3. statestored_8645_20210317-045247.txt
          173 kB
          Andrew Sherman
        4. impalad_8754_20210317-045247.txt
          29 kB
          Andrew Sherman
        5. namenode_10515_jstack_20210317-045247.txt
          41 kB
          Andrew Sherman
        6. hms_16762_jstack_20210317-045247.txt
          291 kB
          Andrew Sherman
        7. impalad_8744_20210317-045247.txt.gz
          48 kB
          Andrew Sherman
        8. impalad_8747_20210317-045247.txt.gz
          37 kB
          Andrew Sherman

          Issue Links

            Activity

              People

              • Assignee:
                asherman Andrew Sherman
                Reporter:
                asherman Andrew Sherman
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: