Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-10416

Getting error "message" : "The KMS key does not contain a location." while using withKmsKey() option at runtime in dataflow job

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Triage Needed
    • Priority: P3
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
    • Environment:
      windows 10 (dataflow runner)

      Description

       
      I have dataflow job that is written using apache beam.Here I am loading the data from one table to another table the mode of table is write truncate and the table contains sensitive data so table configured with kms key. I am following this code and trying to fetch the kms key associated with table and setting this key at the time of (BigQueryIO.write) while writing data into table.

      https://github.com/apache/beam/blob/49403c00499f7f87c2bf9a4c63dec8ebd68d640d/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java#L91-L100

       

       

      {{finalOutput.apply("Write success rows to Sensitive BigQuery",
      BigQueryIO.writeTableRows().withoutValidation()
      .to(options.getTargetTable())
      .withKmsKey(NestedValueProvider.of(options.getTargetTable(),new FetchingKMSKey(options.getProject())).toString()).withoutValidation()
      .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER)
      .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_TRUNCATE));}}

       

       

      {{public class FetchingKMSKey implements SerializableFunction<String, String>

      {

      private static final BigqueryClient BQ_CLIENT = new BigqueryClient("dataMainPipeline");

      public static final Logger LOG = LoggerFactory.getLogger(FetchingKMSKey.class);
      Table table ;
      String tableName;
      String project;
      public FetchingKMSKey(String project)

      { this.project=project; }

      public String apply(String tableName)
      {
      String[] name= tableName.split(Pattern.quote("."));

      try

      { table = BQ_CLIENT.getTableResource(project,name[0],name[1]); }

      catch (IOException e)

      { LOG.error(String.format("exception occured: %s", e.getMessage())); }

      catch (InterruptedException e)

      { LOG.error(String.format("exception occured: %s", e.getMessage())); }

      String kmsKey =table.getEncryptionConfiguration().getKmsKeyName();
      return kmsKey;

      }

      }}}

      {{}}

       

      After creating a template when I am running the job I am getting error.

       

      {{"location" : "US",

      "errors" : [

      { "message" : "The KMS key does not contain a location.", "reason" : "invalid" }

      ],}}

       

      I am completely stuck. anyone could help?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              user-a27 Arzoo Khan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: