Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2925

ToolTest.TestLoadgenZeroRowsPerThread didn't write to the WAL in time

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.11.0
    • Fix Version/s: None
    • Component/s: consensus, log, test
    • Labels:
      None

      Description

      I saw a run of ToolTest.TestLoadgenZeroRowsPerThread that failed because writing to the WAL took 90s. It's not clear why, or whether this was due to hardware issues or a deadlock somewhere.

       

       I0819 03:23:52.093103  7689 catalog_manager.cc:3471] Sending DeleteTablet(TABLET_DATA_DELETED) for tablet 1591b2416d5640d4a1f089ed959128c8 on a3a91c5fde7c49bcb22a8c908d76d428 (127.7.112.193:32983) (Table deleted at 2019-08-19 03:21:51 UTC)
      W0819 03:23:52.271399  7684 debug-util.cc:397] Leaking SignalData structure 0x1fdb4e0 after lost signal to thread 7985
      W0819 03:23:52.271497  7684 kernel_stack_watchdog.cc:198] Thread 7985 stuck at /data/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/consensus/log.cc:671 for 89430ms:
      Kernel stack:
      (could not read kernel stack)
      User stack:
      <Timed out: thread did not respond: maybe it is blocking signals>
      W0819 03:23:52.378173  7985 log.cc:669] Time spent T 00000000000000000000000000000000 P ce4f88cf8a264e82852c46587c4dce11: Append to log took a long time: real 90.538s	user 0.000s	sys 0.000s
      I0819 03:23:52.389586  7619 external_mini_cluster.cc:1136] Killing /tmp/dist-test-taskfg0lzN/build/debug/bin/kudu-master with pid 7680
      I0819 03:23:52.394946  7619 test_util.cc:135] -----------------------------------------------
      I0819 03:23:52.395031  7619 test_util.cc:136] Had fatal failures, leaving test files at /tmp/dist-test-taskfg0lzN/test-tmp/kudu-tool-test.3.ToolTest.TestLoadgenZeroRowsPerThread.1566184909971373-7619
      [  FAILED  ] ToolTest.TestLoadgenZeroRowsPerThread (121696 ms)
      

        Attachments

        1. kudu-tool-test.3.txt.gz
          164 kB
          Andrew Wong

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              awong Andrew Wong
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: