[IMPALA-5247] test_kudu_col_null_changed flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: Impala 2.9.0
• Fix Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: broken-build flaky
• Environment: ec2, c4 instance, Ubuntu 14.04
• Epic Color: ghx-label-7
• Assignee: Matthew Jacobs
• Reporter: Jim Apple
• Created: 22/Apr/17 16:42
• Updated: 18/May/17 22:34
• Resolved: 18/May/17 22:34

Description

Th following failed in what looks like a flaky test to me on the pre-commit run of an unrelated change:

http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/1188/consoleFull

04:26:12 FAIL query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_null_changed
04:26:12 =================================== FAILURES ===================================
04:26:12 ________________ TestKuduOperations.test_kudu_col_null_changed _________________
04:26:12 [gw7] linux2 -- Python 2.7.6 /home/ubuntu/Impala/bin/../infra/python/env/bin/python
04:26:12 query_test/test_kudu.py:209: in test_kudu_col_null_changed
04:26:12     session.flush()
04:26:12 kudu/client.pyx:1190: in kudu.client.Session.flush (kudu/client.cpp:14633)
04:26:12     ???
04:26:12 kudu/errors.pyx:62: in kudu.errors.check_status (kudu/errors.cpp:1073)
04:26:12     ???
04:26:12 E   KuduBadStatus: IO error: Some errors occurred
04:26:12 ---------------------------- Captured stderr setup -----------------------------
04:26:12 MainThread: Using database l28359 as default
04:26:12 SET sync_ddl=False;
04:26:12 -- executing against localhost:21000
04:26:12 DROP DATABASE IF EXISTS `test_kudu_col_null_changed_bc507455` CASCADE;
04:26:12 
04:26:12 SET sync_ddl=False;
04:26:12 -- executing against localhost:21000
04:26:12 CREATE DATABASE `test_kudu_col_null_changed_bc507455`;
04:26:12 
04:26:12 MainThread: Created database "test_kudu_col_null_changed_bc507455" for test ID "query_test/test_kudu.py::TestKuduOperations::()::test_kudu_col_null_changed"
04:26:12 ----------------------------- Captured stderr call -----------------------------
04:26:12 W0422 11:08:21.155036 55261 batcher.cc:325] Timed out: Failed to write batch of 32 ops to tablet 2bb497696479460199dac69188fee4d3 after 1 attempt(s): Failed to write to server: f6a9a0b761cf46d0a40acd0a9208c257 (ip-172-31-7-130:31200): Write RPC to 127.0.0.1:31200 timed out after 5.000s (SENT)
04:26:12 W0422 11:08:21.705691 55261 outbound_call.cc:224] RPC callback for RPC call kudu.tserver.TabletServerService.Write -> {remote=127.0.0.1:31200, user_credentials={real_user=ubuntu}} blocked reactor thread for 550653us