Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-3065

Fine-grained logging in multi-tenant context

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Core Framework
    • Labels:
      None

      Description

      In a multi-tenant approach, it could be interesting to offer to users a way to have a fine-grained logging capabilities per tenant.

      Let's say that root process group is used to have one process group per "project" accessed by different teams/users, it could be really useful to have a log file per process group containing logs for all elements inside this process group.

      A first version could be to develop a custom log appender that would log each message in a file dedicated to its process group parent.

      This improvement will ease the logging management when there are hundreds of processors separated between different teams and this will also prevent the access to the logs concerning one team by another team (could be a security issue).

        Activity

        Hide
        pvillard Pierre Villard added a comment -

        Working on this one and have a couple of questions.

        I started implementing a solution exposing component ID and parent process group ID though MDC feature in SimpleProcessLogger class. I've something working for processors but it remains the case where the parent process group ID changes when a processor is moved from a process group to another. The main issue here (as far as I looked into the code) is: I can catch the update in the StandardProcessorNode class (and update the logger of the node) but I can't access the logger of the processor. I see a couple of options:

        • Expose the component logger in ConfigurableComponent interface but isn't it something to avoid?
        • Have the same logger for both the ProcessorNode and the Processor. But it would still need to expose the logger somehow (Processor interface or ConfigurableComponent). Or I could instantiate the ComponentLogger without the reference to the component in the constructor and add this reference afterwards.

        Before changing a lot of things in the code, feedback would be greatly appreciated

        Also, regarding ReportingTask and ControllerService, I believe this can not be associated to a single parent process group and logs generated by such components should remain in a global log file (or in log file based on component ID). Correct?

        Show
        pvillard Pierre Villard added a comment - Working on this one and have a couple of questions. I started implementing a solution exposing component ID and parent process group ID though MDC feature in SimpleProcessLogger class. I've something working for processors but it remains the case where the parent process group ID changes when a processor is moved from a process group to another. The main issue here (as far as I looked into the code) is: I can catch the update in the StandardProcessorNode class (and update the logger of the node) but I can't access the logger of the processor. I see a couple of options: Expose the component logger in ConfigurableComponent interface but isn't it something to avoid? Have the same logger for both the ProcessorNode and the Processor . But it would still need to expose the logger somehow ( Processor interface or ConfigurableComponent ). Or I could instantiate the ComponentLogger without the reference to the component in the constructor and add this reference afterwards. Before changing a lot of things in the code, feedback would be greatly appreciated Also, regarding ReportingTask and ControllerService , I believe this can not be associated to a single parent process group and logs generated by such components should remain in a global log file (or in log file based on component ID). Correct?
        Hide
        pvillard Pierre Villard added a comment -

        Working on this one and have a couple of questions.

        I started implementing a solution exposing component ID and parent process group ID though MDC feature in SimpleProcessLogger class. I've something working for processors but it remains the case where the parent process group ID changes when a processor is moved from a process group to another. The main issue here (as far as I looked into the code) is: I can catch the update in the StandardProcessorNode class (and update the logger of the node) but I can't access the logger of the processor. I see a couple of options:

        • Expose the component logger in ConfigurableComponent interface but isn't it something to avoid?
        • Have the same logger for both the ProcessorNode and the Processor. But it would still need to expose the logger somehow (Processor interface or ConfigurableComponent). Or I could instantiate the ComponentLogger without the reference to the component in the constructor and add this reference afterwards.

        Before changing a lot of things in the code, feedback would be greatly appreciated

        Also, regarding ReportingTask and ControllerService, I believe this can not be associated to a single parent process group and logs generated by such components should remain in a global log file (or in log file based on component ID). Correct?

        Show
        pvillard Pierre Villard added a comment - Working on this one and have a couple of questions. I started implementing a solution exposing component ID and parent process group ID though MDC feature in SimpleProcessLogger class. I've something working for processors but it remains the case where the parent process group ID changes when a processor is moved from a process group to another. The main issue here (as far as I looked into the code) is: I can catch the update in the StandardProcessorNode class (and update the logger of the node) but I can't access the logger of the processor. I see a couple of options: Expose the component logger in ConfigurableComponent interface but isn't it something to avoid? Have the same logger for both the ProcessorNode and the Processor . But it would still need to expose the logger somehow ( Processor interface or ConfigurableComponent ). Or I could instantiate the ComponentLogger without the reference to the component in the constructor and add this reference afterwards. Before changing a lot of things in the code, feedback would be greatly appreciated Also, regarding ReportingTask and ControllerService , I believe this can not be associated to a single parent process group and logs generated by such components should remain in a global log file (or in log file based on component ID). Correct?

          People

          • Assignee:
            pvillard Pierre Villard
            Reporter:
            pvillard Pierre Villard
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development