[BEAM-12739] Flink JobServer should bundle AWS IO libraries to support staging on S3

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.31.0
• Fix Version/s: 2.33.0
• Component/s: jobserver
• Assignee: Jeremy Lewi
• Reporter: Jeremy Lewi
• Created: 11/Aug/21 00:07
• Updated: 16/Aug/21 18:26
• Resolved: 16/Aug/21 18:26

Description

It doesn't look like the AWS Filesystem libraries are included in the Flink JobServer.

There is a TODO in the code to update the dependencies to support the AWS filesystem.

https://github.com/apache/beam/blob/master/runners/flink/job-server/flink_job_server.gradle#L89

As reported in this mailing list thread trying to use S3 to stage artifacts results in the exception

Aug 06, 2021 1:52:51 AM
org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2
finishStaging

SEVERE: Error staging artifacts

java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: No filesystem found for scheme s3

at java.util.concurrent.FutureTask.report(FutureTask.java:122)

at java.util.concurrent.FutureTask.get(FutureTask.java:192)
 

 

Issue Links

• links to: GitHub Pull Request #15313