Details
-
Improvement
-
Status: Open
-
P3
-
Resolution: Unresolved
-
None
-
None
-
None
Description
I'm using Apache Beam (Java SDK) to insert record in BigQuery using Batch load method (File loads). I want to retrieve those records which failed during insertion.
Is it possible to have a retry policy on failed records?
Actually getting error after retying 1000 times.
Below is my code:
{{}}
public static void insertToBigQueryDataLake( final PCollectionTuple dataStoresCollectionTuple, final TupleTag<KV<DataLake, PayloadSpecs>> dataLakeValidTag, final Long loadJobTriggerFrequency, final Integer loadJobNumShard) { WriteResult writeResult = dataStoresCollectionTuple .get(dataLakeValidTag) .apply(TRANSFORMATION_NAME, DataLakeTableProcessor.dataLakeTableProcessorTransform()) .apply( WRITING_EVENTS_NAME, BigQueryIO.<KV<DataLake, TableRowSpecs>>write() .withMethod(BigQueryIO.Write.Method.FILE_LOADS) .withTriggeringFrequency(Duration.standardMinutes(loadJobTriggerFrequency)) .withNumFileShards(loadJobNumShard) .to(new DynamicTableRowDestinations<>(IS_DATA_LAKE)) .withFormatFunction(BigQueryServiceImpl::dataLakeTableRow)); writeResult.getFailedInserts().apply(ParDo.of(new DoFn<TableRow, Void>() { @ProcessElement public void processElement(final ProcessContext processContext) throws IOException { System.out.println("Table Row : " + processContext.element().toPrettyString()); } })); }
{{}}