[FLINK-5828] BlobServer create cache dir has concurrency safety problem

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.1, 1.3.0
• Assignee: Unassigned
• Reporter: ZhengBowen
• Created: 17/Feb/17 09:32
• Updated: 28/Feb/19 13:20
• Resolved: 20/Feb/17 01:09

Description

Caused by: java.lang.RuntimeException: org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Could not upload the jar files to the job manager.
at FlinkJob_20170217_161058_000004.bind(FlinkJob_20170217_161058_000004.java:45)
at com.aliyun.kepler.rc.query.schedule.FlinkQueryJob.call(FlinkQueryJob.java:53)
at com.aliyun.kepler.rc.query.schedule.FlinkQueryJob.call(FlinkQueryJob.java:13)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.AbstractExecutorService$2.run(AbstractExecutorService.java:120)
... 3 common frames omitted
Caused by: org.apache.flink.client.program.ProgramInvocationException: The program execution failed: Could not upload the jar files to the job manager.
at com.aliyun.kepler.rc.flink.client.Client.runBlocking(Client.java:178)
at org.apache.flink.api.java.ClientEnvironment.execute(ClientEnvironment.java:169)
at org.apache.flink.api.java.ClientEnvironment.execute(ClientEnvironment.java:225)
at FlinkJob_20170217_161058_000004.bind(FlinkJob_20170217_161058_000004.java:42)
... 7 common frames omitted
Caused by: org.apache.flink.runtime.client.JobSubmissionException: Could not upload the jar files to the job manager.
at org.apache.flink.runtime.client.JobClientActor$2.call(JobClientActor.java:359)
at akka.dispatch.Futures$$anonfun$future$1.apply(Future.scala:94)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41)
... 3 common frames omitted
Caused by: java.io.IOException: Could not retrieve the JobManager's blob port.
at org.apache.flink.runtime.blob.BlobClient.uploadJarFiles(BlobClient.java:706)
at org.apache.flink.runtime.jobgraph.JobGraph.uploadUserJars(JobGraph.java:556)
at org.apache.flink.runtime.client.JobClientActor$2.call(JobClientActor.java:357)
... 7 common frames omitted
Caused by: java.io.IOException: PUT operation failed: Server side error: Could not create cache directory '/home/kepler/kepler3012/data/work/blobs/blobStore-c3566cb2-b3d6-40ae-bdcf-594a81c8881b/cache'.
at org.apache.flink.runtime.blob.BlobClient.putInputStream(BlobClient.java:476)
at org.apache.flink.runtime.blob.BlobClient.put(BlobClient.java:338)
at org.apache.flink.runtime.blob.BlobClient.uploadJarFiles(BlobClient.java:730)
at org.apache.flink.runtime.blob.BlobClient.uploadJarFiles(BlobClient.java:701)
... 9 common frames omitted

Issue Links

• links to: GitHub Pull Request #3342