Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4274

Jenkins builds can hang at the BE test "buffered-block-mgr-test"

    Details

      Description

      The umbrella-bulid-and-test job has gotten stuck more than once lately on the BE test buffered-block-mgr-test (as has the ASF "repeated runs" Jenkins job.) A quick JIRA search shows that this test seems to have a history of flakiness. In this latest incarnation, the hang seems to be here:

      [root@impala-boost-static-burst-slave-17c6 ~]# strace -p 21506
      Process 21506 attached - interrupt to quit
      futex(0x4e10244, FUTEX_WAIT_PRIVATE, 1, NULL
      

      There are msgs in the tests WARNING and ERROR output logs, but these same messages show up in the logs of previous tests that have passed, so I don't know how relevant they are. (E.g., maybe there are negative tests that produces these msgs.) Nonetheless, I'll include them:

      WARNINGS:

      Log file created at: 2016/10/03 06:37:11
      Running on machine: impala-boost-static-burst-slave-17c6.vpc.cloudera.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      W1003 06:37:11.796332  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:12.303864  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:12.558909  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:12.796193  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:13.049844  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:13.269994  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:13.496836  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:13.850524  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:14.041249  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:14.301388  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:14.459806  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:14.617405  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      E1003 06:37:15.018292  6468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183': fopen(/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183, "rb+") failed with errno=2 description=Error(2): No such file or directory
      E1003 06:37:15.018648  6466 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183': fopen(/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183, "rb+") failed with errno=2 description=Error(2): No such file or directory
      W1003 06:37:15.102265  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:15.253690  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      E1003 06:37:15.334065  6636 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128': fopen(/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128, "rb+") failed with errno=2 description=Error(2): No such file or directory
      E1003 06:37:15.334220  6635 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128': fopen(/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128, "rb+") failed with errno=2 description=Error(2): No such file or directory
      W1003 06:37:15.410234  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      E1003 06:37:15.495048  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2': Truncate file /tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2 to length 2048 failed with errno 2 (Error(2): No such file or directory)
      W1003 06:37:15.495100  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2': Truncate file /tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2 to length 2048 failed with errno 2 (Error(2): No such file or directory). Will try another scratch file.
      W1003 06:37:15.558523  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      E1003 06:37:15.630766  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied
      W1003 06:37:15.630825  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
      E1003 06:37:15.631402  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied
      W1003 06:37:15.644335  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
      E1003 06:37:15.649292  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied
      W1003 06:37:15.649369  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
      E1003 06:37:15.650197  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied
      W1003 06:37:15.650451  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
      W1003 06:37:15.717438  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      E1003 06:37:15.787549  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied
      W1003 06:37:15.787639  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
      E1003 06:37:15.787919  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied
      W1003 06:37:15.788485  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
      E1003 06:37:15.788746  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied
      W1003 06:37:15.788976  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
      E1003 06:37:15.789222  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied
      W1003 06:37:15.789587  5468 buffered-block-mgr.cc:826] Error while allocating range in scratch file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied. Will try another scratch file.
      W1003 06:37:15.844924  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:15.910274  5468 tmp-file-mgr.cc:74] Running without spill to disk: no scratch directories provided.
      W1003 06:37:15.965059  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:16.083060  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:16.205215  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:16.322567  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      W1003 06:37:16.439827  5468 HiveConf.java:2762] HiveConf of name hive.access.conf.url does not exist
      E1003 06:37:16.502789  5468 buffered-block-mgr-test.cc:410]  Iteration 0
      E1003 06:37:16.817687  5468 buffered-block-mgr-test.cc:410]  Iteration 0
      E1003 06:37:17.006023  5468 buffered-block-mgr-test.cc:410]  Iteration 0
      [etc...]
      

      ERRORS:

      Log file created at: 2016/10/03 06:37:15
      Running on machine: impala-boost-static-burst-slave-17c6.vpc.cloudera.com
      Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
      E1003 06:37:15.018292  6468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183': fopen(/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183, "rb+") failed with errno=2 description=Error(2): No such file or directory
      E1003 06:37:15.018648  6466 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183': fopen(/tmp/impala-scratch/0:0_d67923d0-7996-4667-a081-203e39103183, "rb+") failed with errno=2 description=Error(2): No such file or directory
      E1003 06:37:15.334065  6636 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128': fopen(/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128, "rb+") failed with errno=2 description=Error(2): No such file or directory
      E1003 06:37:15.334220  6635 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128': fopen(/tmp/impala-scratch/0:0_a454c713-c6ce-498a-85d0-0e868ea19128, "rb+") failed with errno=2 description=Error(2): No such file or directory
      E1003 06:37:15.495048  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2': Truncate file /tmp/impala-scratch/0:0_ac4ccd81-6498-4149-b325-261c3efe4af2 to length 2048 failed with errno 2 (Error(2): No such file or directory)
      E1003 06:37:15.630766  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied
      E1003 06:37:15.631402  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_30d7b836-21de-4d58-bf84-2dabc46945a1 failed with errno=13 description=Error(13): Permission denied
      E1003 06:37:15.649292  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied
      E1003 06:37:15.650197  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:1_065537ec-1680-4316-86af-ec5d82c09ac4 failed with errno=13 description=Error(13): Permission denied
      E1003 06:37:15.787549  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied
      E1003 06:37:15.787919  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied
      E1003 06:37:15.788746  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16': Create file /tmp/buffered-block-mgr-test.1/impala-scratch/0:0_50000b9f-89dd-437f-9502-18916cc8da16 failed with errno=13 description=Error(13): Permission denied
      E1003 06:37:15.789222  5468 tmp-file-mgr.cc:258] Error for temporary file '/tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148': Create file /tmp/buffered-block-mgr-test.0/impala-scratch/0:0_cc223ca3-3641-45e4-afa7-149df7649148 failed with errno=13 description=Error(13): Permission denied
      E1003 06:37:16.502789  5468 buffered-block-mgr-test.cc:410]  Iteration 0
      E1003 06:37:16.817687  5468 buffered-block-mgr-test.cc:410]  Iteration 0
      E1003 06:37:17.006023  5468 buffered-block-mgr-test.cc:410]  Iteration 0
      [etc...]
      

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              dknupp David Knupp
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: