[BEAM-6923] OOM errors in jobServer when using GCS artifactDir

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.17.0
• Component/s: sdk-java-harness
• Labels: Done
• Assignee: Ankur Goenka
• Reporter: Lukasz Gajowy
• Created: 27/Mar/19 16:42
• Updated: 06/Oct/20 21:56
• Resolved: 04/Oct/19 18:18

Description

When starting jobServer with artifactDir pointing to a GCS bucket: 

./gradlew :beam-runners-flink_2.11-job-server:runShadow -PflinkMasterUrl=localhost:8081 -PartifactsDir=gs://the-bucket

and running a Java portable pipeline with the following, portability related pipeline options: 

--runner=PortableRunner --jobEndpoint=localhost:8099 --defaultEnvironmentType=DOCKER --defaultEnvironmentConfig=gcr.io/<my-freshly-built-sdk-harness-image>/java:latest'

 

I'm facing a series of OOM errors, like this: 

Exception in thread "grpc-default-executor-3" java.lang.OutOfMemoryError: Java heap space
at com.google.api.client.googleapis.media.MediaHttpUploader.buildContentChunk(MediaHttpUploader.java:606)
at com.google.api.client.googleapis.media.MediaHttpUploader.resumableUpload(MediaHttpUploader.java:408)
at com.google.api.client.googleapis.media.MediaHttpUploader.upload(MediaHttpUploader.java:336)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:508)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:549)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel$UploadOperation.call(AbstractGoogleAsyncWriteChannel.java:301)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

 

This does not happen when I'm using a local filesystem for the artifact staging location. 

 

Issue Links

• links to: GitHub Pull Request #9647