[IMPALA-9710] Flakiness in test_write_error_failover

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 4.0.0
• Labels: broken-build flaky-test
• Epic Color: ghx-label-10
• Assignee: Tim Armstrong
• Reporter: Bikramjeet Vig
• Created: 30/Apr/20 03:17
• Updated: 17/Nov/20 21:27
• Resolved: 17/Nov/20 21:27

Description

Happened in two exhaustive builds so far. test_write_error_failover failed during startup of impalads. One of the impalads failed to find a scratch dir that was created and failed to start up the cluster. Not sure whats going on since other impalads seem to find it.

Impalad 1:

I0428 12:07:38.751999 26356 tmp-file-mgr.cc:229] Using scratch directory /tmp/tmphSI9hn/impala-scratch on disk 0 limit: 8589934592.00 GB
I0428 12:07:38.755487 26356 tmp-file-mgr.cc:229] Using scratch directory /tmp/tmpLmUmqS/impala-scratch on disk 0 limit: 8589934592.00 GB
I0428 12:07:38.762490 26356 tmp-file-mgr.cc:229] Using scratch directory /tmp/tmpWVRtdg/impala-scratch on disk 0 limit: 8589934592.00 GB

Impalad 2:

I0428 12:07:38.789165 26372 tmp-file-mgr.cc:229] Using scratch directory /tmp/tmphSI9hn/impala-scratch on disk 0 limit: 8589934592.00 GB
I0428 12:07:38.791739 26372 tmp-file-mgr.cc:229] Using scratch directory /tmp/tmpLmUmqS/impala-scratch on disk 0 limit: 8589934592.00 GB
I0428 12:07:38.794025 26372 tmp-file-mgr.cc:229] Using scratch directory /tmp/tmpWVRtdg/impala-scratch on disk 0 limit: 8589934592.00 GB

Impalad 3:

I0428 12:07:38.785050 26378 tmp-file-mgr.cc:229] Using scratch directory /tmp/tmphSI9hn/impala-scratch on disk 0 limit: 8589934592.00 GB
I0428 12:07:38.788661 26378 tmp-file-mgr.cc:229] Using scratch directory /tmp/tmpLmUmqS/impala-scratch on disk 0 limit: 8589934592.00 GB
I0428 12:07:38.831136 26378 status.cc:129] Failed to get post-punch file size: Not found: /tmp/tmpLmUmqS/impala-scratch/0f43554c36193ba8:b4d64352fff5389d: No such file or directory (error 2)

Issue Links

• duplicates: IMPALA-9798 TestScratchDir.test_multiple_dirs fails to start impalad