Uploaded image for project: 'Log4j 2'
  1. Log4j 2
  2. LOG4J2-1694

Allow to easily add fields with fixed values to JSON output

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.7, 2.8, 2.8.1, 2.8.2
    • Fix Version/s: None
    • Component/s: Layouts
    • Labels:

      Description

      The Logback JSON Encoder has a feature which allows to specify JSON fields with fixed values. Link: https://github.com/logstash/logstash-logback-encoder#loggingevent_custom_global

      Example:

      <encoder class="net.logstash.logback.encoder.LogstashEncoder">
        <customFields>{"appname":"myWebservice","roles":["customerorder","auth"],"buildinfo":{"version":"Version 0.1.0-SNAPSHOT","lastcommit":"75473700d5befa953c45f630c6d9105413c16fe1"}}</customFields>
      </encoder>
      

      This is a very convenient way to specify fixed attributes like application, service or environment.

      So far I don't see such a possibility in log4j.

        Issue Links

          Activity

          Hide
          garydgregory Gary Gregory added a comment -

          Patches welcome

          Show
          garydgregory Gary Gregory added a comment - Patches welcome
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          GelfLayout has a similar possibility.

          Show
          mikaelstaldal Mikael Ståldal added a comment - GelfLayout has a similar possibility.
          Hide
          mbilgen Melike B Bilgen added a comment -

          Can I take this issue?

          Show
          mbilgen Melike B Bilgen added a comment - Can I take this issue?
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          I would like to know more about the use-case, and whether GelfLayout can fulfil it.

          Show
          mikaelstaldal Mikael Ståldal added a comment - I would like to know more about the use-case, and whether GelfLayout can fulfil it.
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          Even if the Gelf Layout has the feature why not also add key/value pair support to the JSON Layout?

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - Even if the Gelf Layout has the feature why not also add key/value pair support to the JSON Layout?
          Hide
          mikaelstaldal Mikael Ståldal added a comment - - edited

          I think we should have a bit of resistance against feature creep. Before adding a feature like this, I would like to understand the use-case, and whether that use-case can be fulfilled with another, already existing, feature.

          And if we add this to JsonLayout, should we also add it to XML and YAML layouts?

          Show
          mikaelstaldal Mikael Ståldal added a comment - - edited I think we should have a bit of resistance against feature creep. Before adding a feature like this, I would like to understand the use-case, and whether that use-case can be fulfilled with another, already existing, feature. And if we add this to JsonLayout, should we also add it to XML and YAML layouts?
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          Yes, if we add it to JsonLayout it should be added to XML and YAML as well.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - Yes, if we add it to JsonLayout it should be added to XML and YAML as well.
          Hide
          garydgregory Gary Gregory added a comment -

          We have a class called org.apache.logging.log4j.core.layout.LoggerFields which is specific to Syslog but shows that the concept is not new.

          Show
          garydgregory Gary Gregory added a comment - We have a class called org.apache.logging.log4j.core.layout.LoggerFields which is specific to Syslog but shows that the concept is not new.
          Hide
          raimar.falke Raimar Falke added a comment -

          The use case is that each log message has a common set of attributes. Useful are:

          • "environment" with values like prod/live, test, uat, integration, staging3
          • "application" to use a single ELK installation for the whole IT department
          • "service" to split a single application. Values are "user", "shipping", ...

          Application and service are usually fixed for a certain war-file while the environment depends on the deployment.

          In the logback case a single config file is created during the building/deployment process which ensures that each log message has all these attributes. The question is how this could be achieved with log4j.

          One solution we discussed internally was to put these attributes in all MDCs. But it hard to ensure that each Thread is covered. So I could also image a solution which doesn't add this feature to a or multiple Layouts but which changes the way the LoggingEvent is created. So that you can specify a fixed base set of properties each LoggingEvent will have.

          Show
          raimar.falke Raimar Falke added a comment - The use case is that each log message has a common set of attributes. Useful are: "environment" with values like prod/live, test, uat, integration, staging3 "application" to use a single ELK installation for the whole IT department "service" to split a single application. Values are "user", "shipping", ... Application and service are usually fixed for a certain war-file while the environment depends on the deployment. In the logback case a single config file is created during the building/deployment process which ensures that each log message has all these attributes. The question is how this could be achieved with log4j. One solution we discussed internally was to put these attributes in all MDCs. But it hard to ensure that each Thread is covered. So I could also image a solution which doesn't add this feature to a or multiple Layouts but which changes the way the LoggingEvent is created. So that you can specify a fixed base set of properties each LoggingEvent will have.
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          Thanks. That is a valid use case. I have a similar use case myself, and use GelfLayout which has this functionallity:

                  <GelfLayout host="user">
                      <KeyValuePair key="environment" value="prod"/>
                      <KeyValuePair key="application" value="whatever"/>
                  </GelfLayout>
          

          So I guess you would like something similar for JsonLayout?

          Maybe you can use GelfLayout instead of JsonLayout? https://www.elastic.co/guide/en/logstash/current/plugins-inputs-gelf.html

          Show
          mikaelstaldal Mikael Ståldal added a comment - Thanks. That is a valid use case. I have a similar use case myself, and use GelfLayout which has this functionallity: <GelfLayout host= "user" > <KeyValuePair key= "environment" value= "prod" /> <KeyValuePair key= "application" value= "whatever" /> </GelfLayout> So I guess you would like something similar for JsonLayout? Maybe you can use GelfLayout instead of JsonLayout? https://www.elastic.co/guide/en/logstash/current/plugins-inputs-gelf.html
          Hide
          garydgregory Gary Gregory added a comment -

          We could also give the AbstractStringLayout an abstract builder and add a KeyValuePair to it.

          Then all AbstractStringLayout subclasses can use this new builder.

          Gary

          Show
          garydgregory Gary Gregory added a comment - We could also give the AbstractStringLayout an abstract builder and add a KeyValuePair to it. Then all AbstractStringLayout subclasses can use this new builder. Gary
          Hide
          gpolaert gpolaert added a comment -

          Hello,
          IMO, allow user to add `<KeyValuePair>` is really useful.
          So, is someone assigned to this issue? If no, I can work on it and submit a PR if you want.

          Show
          gpolaert gpolaert added a comment - Hello, IMO, allow user to add `<KeyValuePair>` is really useful. So, is someone assigned to this issue? If no, I can work on it and submit a PR if you want.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9383ab7c2fafce6005df6e97ee25ae700b71502a in logging-log4j2's branch refs/heads/LOG4J2-1694 from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=9383ab7 ]

          LOG4J2-1694 Update changes

          Show
          jira-bot ASF subversion and git services added a comment - Commit 9383ab7c2fafce6005df6e97ee25ae700b71502a in logging-log4j2's branch refs/heads/ LOG4J2-1694 from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=9383ab7 ] LOG4J2-1694 Update changes
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit b2e2d06d23cc6c5d1e5da7857f9b2cd52e95ab5e in logging-log4j2's branch refs/heads/LOG4J2-1694 from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=b2e2d06 ]

          LOG4J2-1694 Add builds to XmlLayout and YamlLayout, add additionalFields property to JsonLayout, XmlLayout and YamlLayout (currently no-op, W.I.P.)

          Show
          jira-bot ASF subversion and git services added a comment - Commit b2e2d06d23cc6c5d1e5da7857f9b2cd52e95ab5e in logging-log4j2's branch refs/heads/ LOG4J2-1694 from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=b2e2d06 ] LOG4J2-1694 Add builds to XmlLayout and YamlLayout, add additionalFields property to JsonLayout, XmlLayout and YamlLayout (currently no-op, W.I.P.)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 0166ac20a1d3dc154e1aa16a12fa0aeb990b5e68 in logging-log4j2's branch refs/heads/LOG4J2-1694 from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=0166ac2 ]

          LOG4J2-1694 Fix formatting

          Show
          jira-bot ASF subversion and git services added a comment - Commit 0166ac20a1d3dc154e1aa16a12fa0aeb990b5e68 in logging-log4j2's branch refs/heads/ LOG4J2-1694 from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=0166ac2 ] LOG4J2-1694 Fix formatting
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1c2ad7b807c3a34039206e17f3f71be6b6dc5216 in logging-log4j2's branch refs/heads/LOG4J2-1694 from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=1c2ad7b ]

          Merge branch 'master' into LOG4J2-1694

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1c2ad7b807c3a34039206e17f3f71be6b6dc5216 in logging-log4j2's branch refs/heads/ LOG4J2-1694 from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=1c2ad7b ] Merge branch 'master' into LOG4J2-1694
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit c09c4b1fec48efab30c4f563df707b25f1cd28b6 in logging-log4j2's branch refs/heads/LOG4J2-1694 from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=c09c4b1 ]

          LOG4J2-1694 Do not use deprecated method

          Show
          jira-bot ASF subversion and git services added a comment - Commit c09c4b1fec48efab30c4f563df707b25f1cd28b6 in logging-log4j2's branch refs/heads/ LOG4J2-1694 from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=c09c4b1 ] LOG4J2-1694 Do not use deprecated method
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit dbd2d252a1b4139a9bd9eb213c89f28498db6dcf in logging-log4j2's branch refs/heads/LOG4J2-1694 from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=dbd2d25 ]

          LOG4J2-1694 Remove additionalFields

          Show
          jira-bot ASF subversion and git services added a comment - Commit dbd2d252a1b4139a9bd9eb213c89f28498db6dcf in logging-log4j2's branch refs/heads/ LOG4J2-1694 from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=dbd2d25 ] LOG4J2-1694 Remove additionalFields
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 9383ab7c2fafce6005df6e97ee25ae700b71502a in logging-log4j2's branch refs/heads/master from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=9383ab7 ]

          LOG4J2-1694 Update changes

          Show
          jira-bot ASF subversion and git services added a comment - Commit 9383ab7c2fafce6005df6e97ee25ae700b71502a in logging-log4j2's branch refs/heads/master from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=9383ab7 ] LOG4J2-1694 Update changes
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit b2e2d06d23cc6c5d1e5da7857f9b2cd52e95ab5e in logging-log4j2's branch refs/heads/master from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=b2e2d06 ]

          LOG4J2-1694 Add builds to XmlLayout and YamlLayout, add additionalFields property to JsonLayout, XmlLayout and YamlLayout (currently no-op, W.I.P.)

          Show
          jira-bot ASF subversion and git services added a comment - Commit b2e2d06d23cc6c5d1e5da7857f9b2cd52e95ab5e in logging-log4j2's branch refs/heads/master from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=b2e2d06 ] LOG4J2-1694 Add builds to XmlLayout and YamlLayout, add additionalFields property to JsonLayout, XmlLayout and YamlLayout (currently no-op, W.I.P.)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 0166ac20a1d3dc154e1aa16a12fa0aeb990b5e68 in logging-log4j2's branch refs/heads/master from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=0166ac2 ]

          LOG4J2-1694 Fix formatting

          Show
          jira-bot ASF subversion and git services added a comment - Commit 0166ac20a1d3dc154e1aa16a12fa0aeb990b5e68 in logging-log4j2's branch refs/heads/master from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=0166ac2 ] LOG4J2-1694 Fix formatting
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1c2ad7b807c3a34039206e17f3f71be6b6dc5216 in logging-log4j2's branch refs/heads/master from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=1c2ad7b ]

          Merge branch 'master' into LOG4J2-1694

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1c2ad7b807c3a34039206e17f3f71be6b6dc5216 in logging-log4j2's branch refs/heads/master from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=1c2ad7b ] Merge branch 'master' into LOG4J2-1694
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit c09c4b1fec48efab30c4f563df707b25f1cd28b6 in logging-log4j2's branch refs/heads/master from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=c09c4b1 ]

          LOG4J2-1694 Do not use deprecated method

          Show
          jira-bot ASF subversion and git services added a comment - Commit c09c4b1fec48efab30c4f563df707b25f1cd28b6 in logging-log4j2's branch refs/heads/master from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=c09c4b1 ] LOG4J2-1694 Do not use deprecated method
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit dbd2d252a1b4139a9bd9eb213c89f28498db6dcf in logging-log4j2's branch refs/heads/master from Mikael Ståldal
          [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=dbd2d25 ]

          LOG4J2-1694 Remove additionalFields

          Show
          jira-bot ASF subversion and git services added a comment - Commit dbd2d252a1b4139a9bd9eb213c89f28498db6dcf in logging-log4j2's branch refs/heads/master from Mikael Ståldal [ https://git-wip-us.apache.org/repos/asf?p=logging-log4j2.git;h=dbd2d25 ] LOG4J2-1694 Remove additionalFields
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          I did not find any reasonable way to do this given the way JsonLayout is implemented.

          Anyone else is free to try. Melike B Bilgen gpolaert

          I have made some refactorings and clean-ups in JsonLayout, XmlLayout, YamlLayout and AbstractJacksonLayout which should make it easier to get started with this, they are in Git master now.

          If you want to try implementing this, consider doing it in AbstractJacksonLayout so it can be applied for all three layouts.

          Show
          mikaelstaldal Mikael Ståldal added a comment - I did not find any reasonable way to do this given the way JsonLayout is implemented. Anyone else is free to try. Melike B Bilgen gpolaert I have made some refactorings and clean-ups in JsonLayout, XmlLayout, YamlLayout and AbstractJacksonLayout which should make it easier to get started with this, they are in Git master now. If you want to try implementing this, consider doing it in AbstractJacksonLayout so it can be applied for all three layouts.
          Hide
          vy Volkan Yazıcı added a comment - - edited

          Hey Mikael Ståldal! I've seen your changes on AbstractJacksonLayout. May I ask why did you revert them? What was the problem? If I am not mistaken, you just need to append a .withAttributes(additionalFields) call to JacksonFactory.java:222 where you create the ObjectWriter.

          Show
          vy Volkan Yazıcı added a comment - - edited Hey Mikael Ståldal ! I've seen your changes on AbstractJacksonLayout . May I ask why did you revert them? What was the problem? If I am not mistaken, you just need to append a .withAttributes(additionalFields) call to JacksonFactory.java:222 where you create the ObjectWriter .
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          I was not aware of any such feature in Jackson. Do you want to provide a patch? You can base it on my reverted changes if you want.

          Show
          mikaelstaldal Mikael Ståldal added a comment - I was not aware of any such feature in Jackson. Do you want to provide a patch? You can base it on my reverted changes if you want.
          Hide
          vy Volkan Yazıcı added a comment -

          Mikael Ståldal, I'm already working on it. Though I cannot see a LOG4J2-1694 branch on master at GitHub. Anyway... I'll first finish my draft and see how I can push/merge it back to origin in a separate branch.

          Show
          vy Volkan Yazıcı added a comment - Mikael Ståldal , I'm already working on it. Though I cannot see a LOG4J2-1694 branch on master at GitHub. Anyway... I'll first finish my draft and see how I can push/merge it back to origin in a separate branch.
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          Volkan Yazıcı I have now recreated the LOG4J2-1694 branch. I tried to used withAttributes, but it doesn't seem to work. Feel free to have a try.

          Show
          mikaelstaldal Mikael Ståldal added a comment - Volkan Yazıcı I have now recreated the LOG4J2-1694 branch. I tried to used withAttributes , but it doesn't seem to work. Feel free to have a try.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user mdvorak commented on the issue:

          https://github.com/apache/logging-log4j2/pull/110

          Link to JIRA issue:
          https://issues.apache.org/jira/browse/LOG4J2-1694

          Show
          githubbot ASF GitHub Bot added a comment - Github user mdvorak commented on the issue: https://github.com/apache/logging-log4j2/pull/110 Link to JIRA issue: https://issues.apache.org/jira/browse/LOG4J2-1694

            People

            • Assignee:
              Unassigned
              Reporter:
              raimar.falke Raimar Falke
            • Votes:
              2 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:

                Development