[BEAM-8089] Error while using customGcsTempLocation() with Dataflow

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Affects Version/s: 2.13.0
• Component/s: io-java-gcp
• Assignee: Unassigned
• Reporter: Harshit Dwivedi
• Created: 25/Aug/19 12:02
• Updated: 04/Jun/22 12:43

Description

I have the following code snippet which writes content to BigQuery via File Loads.

Currently the files are being written to a GCS Bucket, but I want to write them to the local file storage of Dataflow instead and want BigQuery to load data from there.

 

 

 

BigQueryIO
 .writeTableRows()
 .withNumFileShards(100)
 .withTriggeringFrequency(Duration.standardSeconds(90))
 .withMethod(BigQueryIO.Write.Method.FILE_LOADS)
 .withSchema(getSchema())
 .withoutValidation()
 .withCustomGcsTempLocation(new ValueProvider<String>() {
    @Override
    public String get(){
         return "/home/harshit/testFiles";     
    }
    @Override
    public boolean isAccessible(){
         return true;     
    }})
 .withTimePartitioning(new TimePartitioning().setType("DAY"))
 .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_IF_NEEDED)
 .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND)
 .to(tableName));

 

 

On running this, I don't see any files being written to the provided path and the BQ load jobs fail with an IOException.

 

I looked at the docs, but I was unable to find any working example for this.