Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-10570

FE tests get stuck and eventually time out during UBSAN build

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Blocker
    • Resolution: Unresolved
    • Affects Version/s: Impala 4.0.0
    • Fix Version/s: None
    • Component/s: Frontend
    • Labels:
    • Epic Color:
      ghx-label-5

      Description

      During UBSAN builds on private infrastructure (using CentOS 7.4 as the OS platform) FE tests get stuck, then eventually get killed by the timeout mechanism in buildall.sh. Unfortunately output buffering makes the Jenkins console log slightly confusing, so it is not easy to tell which exact FE test is getting stuck: the timeout and build shutdown sequences seem to be artificially inserted into the middle of the FE build result summary.

      Representative log section:

      01:44:28.496 [INFO] Tests run: 99, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.305 s - in org.apache.impala.analysis.ParserTest
      01:44:28.496 [INFO] Running org.apache.impala.analysis.ToSqlTest
      01:44:28.496 [INFO] Tests run: 41, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.88 s - in org.apache.impala.analysis.ToSqlTest
      01:44:28.496 [INFO] Running org.apache.impala.analysis.Ana
      01:44:28.496 
      20:55:34.940 **** run-all-tests.sh TIMED OUT! ****
      20:55:34.943 
      20:55:34.943 
      20:55:34.943 **** Generating backtrace of impalad with process id: 15971 to /data/jenkins/workspace/impala-cdpd-master-core-ubsan/repos/Impala/logs/timeout_stacktrace/impalad_15971_20210308-093840.txt ****
      
      [..... lots of debug and strack trace output elided for brevity's sake....]
      
      [.... complete log section for build shutdown elided.....]
      
      20:57:13.167 /data/jenkins/workspace/impala-cdpd-master-core-ubsan/repos/Impala/common/thrift/MetricDefs.thrift created.
      20:57:13.167 /data/jenkins/workspace/impala-cdpd-master-core-ubsan/impala_schema.mdl created.
      20:57:13.167 + pg_dump -U hiveuser HMS_data_jenkins_workspace_impala_cdpd_master_core_ubsan_re_cdp
      20:57:13.167 + exit 1
      20:57:13.502 Process leaked file descriptors. See https://jenkins.io/redirect/troubleshooting/process-leaked-file-descriptors for more information
      20:57:23.505 Build step 'Execute shell' marked build as failure
      20:57:23.576 lyzeStmtsTest
      20:57:27.934 [INFO] Tests run: 67, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.999 s - in org.apache.impala.analysis.AnalyzeStmtsTest
      20:57:27.934 [INFO] Running org.apache.impala.analysis.ExprRewriteRulesTest
      20:57:27.934 [INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.121 s - in org.apache.impala.analysis.ExprRewriteRulesTest
      
      [.... rest of the FE test result summary follows.....]
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              laszlog Laszlo Gaal
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: