Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-8611

GCP BigQuery processors support using designate project resource for ingestion

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.11.4
    • Fix Version/s: None
    • Component/s: Extensions
    • Labels:

      Description

      For now, PutBigQueryBatch processor and PutBigQueryStreaming processor can only assign a single project id for consuming resources and do ingestion. But in some business cases, the project providing resources and the project which be inserted are not always the same. 

      src/main/java/org/apache/nifi/processors/gcp/AbstractGCPProcessor.java

       

      ......
      public static final PropertyDescriptor PROJECT_ID = new PropertyDescriptor
              .Builder().name("gcp-project-id")
              .displayName("Project ID")
              .description("Google Cloud Project ID")
              .required(false)
              .expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY)
              .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
              .build();
      ......

       

      We've test a solution which is workable, which is, adding another property DESIGNATE_PROJECT_ID in AbstractBigQueryProcessor, it'll only impact PutBigQueryBatch processor and PutBigQueryStreaming processor.

      If user provides designate project id:

      • Use PROJECT_ID (defined in AbstractGCPProcessor) as resource consuming project. 
      • Put data into DESIGNATE_PROJECT_ID  (defined in AbstractBigQueryProcessor). 

      If user does not provide designate project id:

      • Use PROJECT_ID (defined in AbstractGCPProcessor) as resource consuming project. 
      • Put data into PROJECT_ID  (defined in AbstractGCPProcessor). 

      Since we already implemented this solution in production environment, I'll submit a PR later for this improvement. 

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                joewitt Joe Witt
                Reporter:
                byakuinss Chih Han Yu
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h