[IMPALA-9165] Precommit jobs getting stuck in testdata/bin/create-hbase.sh

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 3.4.0
• Component/s: Infrastructure
• Labels: broken-build
• Epic Color: ghx-label-2
• Assignee: Joe McDonnell
• Reporter: Joe McDonnell
• Created: 18/Nov/19 18:29
• Updated: 11/Dec/19 19:04
• Resolved: 11/Dec/19 19:04

Description

Several of our precommit jobs have been getting stuck and cancelled with this log:

01:54:19 /home/ubuntu/Impala/testdata/target
01:54:19 SUCCESS, data generated into /home/ubuntu/Impala/testdata/target
01:54:20 Executing: create-load-data.sh 
01:54:20 Generating HBase data (logging to /home/ubuntu/Impala/logs/data_loading/create-hbase.log)... 
11:23:26 FATAL: Unable to delete script file /tmp/jenkins4199210326564796706.sh
11:23:26 java.lang.InterruptedException
11:23:26 	at java.lang.Object.wait(Native Method)
11:23:26 	at hudson.remoting.Request.call(Request.java:177)
11:23:26 	at hudson.remoting.Channel.call(Channel.java:956)
11:23:26 	at hudson.FilePath.act(FilePath.java:1070)
11:23:26 	at hudson.FilePath.act(FilePath.java:1059)
11:23:26 	at hudson.FilePath.delete(FilePath.java:1540)
11:23:26 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:123)
11:23:26 	at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
11:23:26 	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
11:23:26 	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
11:23:26 	at hudson.model.Build$BuildExecution.build(Build.java:206)
11:23:26 	at hudson.model.Build$BuildExecution.doRun(Build.java:163)
11:23:26 	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
11:23:26 	at hudson.model.Run.execute(Run.java:1818)
11:23:26 	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
11:23:26 	at hudson.model.ResourceController.execute(ResourceController.java:97)
11:23:26 	at hudson.model.Executor.run(Executor.java:429)

This indicates that the job is getting stuck inside testdata/bin/create-hbase.sh. The logs are not available, because the job gets cancelled.

Some example jobs with this problem:

https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8847/

https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8762/

https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/8850/