Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-5670

Improve relationship documentation about FF penalization

    XMLWordPrintableJSON

    Details

      Description

      It'd be nice to improve the generated documentation of the relationships to indicate if a flow file is going to be penalized when transferred into a relationship. That could help users not to "self loop" a relationship when flow files are not penalized and to actually better understand observed behaviors.

      As far I can see, we always follow the below pattern:

      flowFile = session.penalize(flowFile);
      session.transfer(flowFile, REL_FAILURE);

      What I suggest is to have something like:

      public static final Relationship REL_FAILURE = new Relationship.Builder()
              .name("failure")
              .description("description")
              .penalize(true)
              .build();

      And, in the 'transfer' method implementation, check if the 'Relationship' has the penalize flag to penalize (or not) the flow file.

      Problem is: we have cases where we route to the same relationship penalized AND non penalized flow files. I'm wondering if that's a valid approach. IMO we penalize a flow file when we expect the relationship to be self looped and when the cause could be resolved later. We penalize the flow file to avoid resource exhaustion by having a processor continually reprocessing the same flow file. However... if we have penalized and non-penalized flow files routed to the same relationship, are we not exposing the users to some risks? Should we have a 'retry' relationship or ensure that we're penalizing all the flow files?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              pvillard Pierre Villard
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: