[IMPALA-4274] Jenkins builds can hang at the BE test "buffered-block-mgr-test"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.7.0
• Fix Version/s: Impala 2.8.0
• Component/s: Backend
• Labels: broken-build test_issue
• Assignee: Tim Armstrong
• Reporter: David Knupp
• Created: 11/Oct/16 19:30
• Updated: 12/Oct/16 15:26
• Resolved: 12/Oct/16 15:26

Description

The umbrella-bulid-and-test job has gotten stuck more than once lately on the BE test buffered-block-mgr-test (as has the ASF "repeated runs" Jenkins job.) A quick JIRA search shows that this test seems to have a history of flakiness. In this latest incarnation, the hang seems to be here:

[root@impala-boost-static-burst-slave-17c6 ~]# strace -p 21506
Process 21506 attached - interrupt to quit
futex(0x4e10244, FUTEX_WAIT_PRIVATE, 1, NULL

There are msgs in the tests WARNING and ERROR output logs, but these same messages show up in the logs of previous tests that have passed, so I don't know how relevant they are. (E.g., maybe there are negative tests that produces these msgs.) Nonetheless, I'll include them:

WARNINGS:

Log file created at: 2016/10/03 06:37:11
Running on machine: impala-boost-static-burst-slave-17c6.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
W1003 06:37:11.796332  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:12.303864  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:12.558909  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:12.796193  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:13.049844  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:13.269994  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:13.496836  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:13.850524  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:14.041249  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:14.301388  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:14.459806  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:14.617405  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
E1003 06:37:15.018292  6468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183': fopen(/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183, "rb+") failed with errno=2 description=Error(2): No such file or directory
E1003 06:37:15.018648  6466 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183': fopen(/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183, "rb+") failed with errno=2 description=Error(2): No such file or directory
W1003 06:37:15.102265  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:15.253690  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
E1003 06:37:15.334065  6636 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128': fopen(/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128, "rb+") failed with errno=2 description=Error(2): No such file or directory
E1003 06:37:15.334220  6635 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128': fopen(/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128, "rb+") failed with errno=2 description=Error(2): No such file or directory
W1003 06:37:15.410234  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
E1003 06:37:15.495048  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2': Truncate file /tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2 to length 2048 failed with errno 2 (Error(2): No such file or directory)
W1003 06:37:15.495100  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2': Truncate file /tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2 to length 2048 failed with errno 2 (Error(2): No such file or directory). Will try another scratch file.
W1003 06:37:15.558523  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
E1003 06:37:15.630766  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied
W1003 06:37:15.630825  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
E1003 06:37:15.631402  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied
W1003 06:37:15.644335  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
E1003 06:37:15.649292  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied
W1003 06:37:15.649369  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
E1003 06:37:15.650197  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied
W1003 06:37:15.650451  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
W1003 06:37:15.717438  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
E1003 06:37:15.787549  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied
W1003 06:37:15.787639  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
E1003 06:37:15.787919  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied
W1003 06:37:15.788485  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
E1003 06:37:15.788746  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied
W1003 06:37:15.788976  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
E1003 06:37:15.789222  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied
W1003 06:37:15.789587  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
W1003 06:37:15.844924  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:15.910274  5468 tmp-file-mgr.cc:74] Running without spill to disk: no scratch directories provided.
W1003 06:37:15.965059  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:16.083060  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:16.205215  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:16.322567  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
W1003 06:37:16.439827  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
E1003 06:37:16.502789  5468 buffered-block-mgr-test.cc:410]  Iteration 0
E1003 06:37:16.817687  5468 buffered-block-mgr-test.cc:410]  Iteration 0
E1003 06:37:17.006023  5468 buffered-block-mgr-test.cc:410]  Iteration 0
[etc...]

ERRORS:

Log file created at: 2016/10/03 06:37:15
Running on machine: impala-boost-static-burst-slave-17c6.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E1003 06:37:15.018292  6468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183': fopen(/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183, "rb+") failed with errno=2 description=Error(2): No such file or directory
E1003 06:37:15.018648  6466 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183': fopen(/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183, "rb+") failed with errno=2 description=Error(2): No such file or directory
E1003 06:37:15.334065  6636 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128': fopen(/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128, "rb+") failed with errno=2 description=Error(2): No such file or directory
E1003 06:37:15.334220  6635 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128': fopen(/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128, "rb+") failed with errno=2 description=Error(2): No such file or directory
E1003 06:37:15.495048  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2': Truncate file /tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2 to length 2048 failed with errno 2 (Error(2): No such file or directory)
E1003 06:37:15.630766  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied
E1003 06:37:15.631402  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied
E1003 06:37:15.649292  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied
E1003 06:37:15.650197  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied
E1003 06:37:15.787549  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied
E1003 06:37:15.787919  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied
E1003 06:37:15.788746  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied
E1003 06:37:15.789222  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied
E1003 06:37:16.502789  5468 buffered-block-mgr-test.cc:410]  Iteration 0
E1003 06:37:16.817687  5468 buffered-block-mgr-test.cc:410]  Iteration 0
E1003 06:37:17.006023  5468 buffered-block-mgr-test.cc:410]  Iteration 0
[etc...]