[NIFI-8611] GCP BigQuery processors support using designate project resource for ingestion

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.11.4
• Component/s: Extensions
• Labels: GCP bigquery
• Assignee: Joe Witt
• Reporter: Chih Han Yu
• Created: 18/May/21 09:29
• Updated: 16/Mar/23 00:17

Description

For now, PutBigQueryBatch processor and PutBigQueryStreaming processor can only assign a single project id for consuming resources and do ingestion. But in some business cases, the project providing resources and the project which be inserted are not always the same. 

src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java

 

......
public static final PropertyDescriptor PROJECT_ID = new PropertyDescriptor
        .Builder().name("gcp-project-id")
        .displayName("Project ID")
        .description("Google Cloud Project ID")
        .required(false)
        .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
        .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
        .build();
......

 

We've test a solution which is workable, which is, adding another property DESIGNATE_PROJECT_ID in AbstractBigQueryProcessor, it'll only impact PutBigQueryBatch processor and PutBigQueryStreaming processor.

If user provides designate project id:

• Use PROJECT_ID (defined in AbstractGCPProcessor) as resource consuming project. 
• Put data into DESIGNATE_PROJECT_ID  (defined in AbstractBigQueryProcessor). 

If user does not provide designate project id:

• Use PROJECT_ID (defined in AbstractGCPProcessor) as resource consuming project. 
• Put data into PROJECT_ID  (defined in AbstractGCPProcessor). 

Since we already implemented this solution in production environment, I'll submit a PR later for this improvement. 

 

Issue Links

• links to: GitHub Pull Request #5135