Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2925

ToolTest.TestLoadgenZeroRowsPerThread didn't write to the WAL in time

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.11.0
    • None
    • consensus, log, test
    • None

    Description

      I saw a run of ToolTest.TestLoadgenZeroRowsPerThread that failed because writing to the WAL took 90s. It's not clear why, or whether this was due to hardware issues or a deadlock somewhere.

       

       I0819 03:23:52.093103  7689 catalog_manager.cc:3471] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1591b2416d5640d4a1f089ed959128c8 on a3a91c5fde7c49bcb22a8c908d76d428 (127.7.112.193:32983) (Table deleted at 2019-08-19 03:21:51 UTC)
      W0819 03:23:52.271399  7684 debug-util.cc:397] Leaking SignalData structure 0x1fdb4e0 after lost signal to thread 7985
      W0819 03:23:52.271497  7684 kernel_stack_watchdog.cc:198] Thread 7985 stuck at /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/consensus/log.cc:671 for 89430ms:
      Kernel stack:
      (could not read kernel stack)
      User stack:
      <Timed out: thread did not respond: maybe it is blocking signals>
      W0819 03:23:52.378173  7985 log.cc:669] Time spent T 00000000000000000000000000000000 P ce4f88cf8a264e82852c46587c4dce11: Append to log took a long time: real 90.538s	user 0.000s	sys 0.000s
      I0819 03:23:52.389586  7619 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskfg0lzN/build/debug/bin/kudu-master with pid 7680
      I0819 03:23:52.394946  7619 test_util.cc:135] -----------------------------------------------
      I0819 03:23:52.395031  7619 test_util.cc:136] Had fatal failures, leaving test files at /tmp/dist-test-taskfg0lzN/test-tmp/kudu-tool-test.3.ToolTest.TestLoadgenZeroRowsPerThread.1566184909971373-7619
      [  FAILED  ] ToolTest.TestLoadgenZeroRowsPerThread (121696 ms)
      

      Attachments

        1. kudu-tool-test.3.txt.gz
          164 kB
          Andrew Wong

        Activity

          People

            Unassigned Unassigned
            awong Andrew Wong
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: