Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-3772

BigQueryIO - Can't use DynamicDestination with CREATE_IF_NEEDED for unbounded PCollection and FILE_LOADS

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.2.0, 2.3.0
    • Fix Version/s: None
    • Component/s: io-java-gcp
    • Labels:
      None
    • Environment:
      Dataflow streaming pipeline

      Description

      My workflow : KAFKA -> Dataflow streaming -> BigQuery

      Given that having low-latency isn't important in my case, I use FILE_LOADS to reduce the costs. I'm using BigQueryIO.Write with a DynamicDestination, which is a table with the current hour as a suffix.

      This BigQueryIO.Write is configured like this :

      .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED)
      .withMethod(Method.FILE_LOADS)
      .withTriggeringFrequency(triggeringFrequency)
      .withNumFileShards(100)
      

      The first table is successfully created and is written to. But then the following tables are never created and I get these exceptions:

      (99e5cd8c66414e7a): java.lang.RuntimeException: Failed to create load job with id prefix 5047f71312a94bf3a42ee5d67feede75_5295fbf25e1a7534f85e25dcaa9f4986_00001_00023, reached max retries: 3, last failed load job: {
        "configuration" : {
          "load" : {
            "createDisposition" : "CREATE_NEVER",
            "destinationTable" : {
              "datasetId" : "dev_mydataset",
              "projectId" : "myproject-id",
              "tableId" : "mytable_20180302_16"
            },
      

      The CreateDisposition used is CREATE_NEVER, contrary as CREATE_IF_NEEDED as specified.

        Attachments

          Activity

            People

            • Assignee:
              jkff Eugene Kirpichov
              Reporter:
              benjben Benjamin BENOIST
            • Votes:
              2 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: