Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      5 August 2015 --The Apache Logging Services™ Project Management Committee (PMC) has announced that the Log4j™ 1.x logging framework has reached its end of life (EOL) and is no longer officially supported.

      https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces

      A whole framework log4j2 upgrade has to be synchronized, partly for improved performance brought about by log4j2.

      https://logging.apache.org/log4j/2.x/manual/async.html#Performance

        Issue Links

          Activity

          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment -

          The above patch should simplify some of the challenges other projects have in moving to Log4j 2 as Hadoop won't require Log4j 1's AppenderSkeleton unless Log4j 1 is being used. It also eliminates the need to exclude the log4j 1.x and slf4j-log4j12 jars since they are now optional.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - The above patch should simplify some of the challenges other projects have in moving to Log4j 2 as Hadoop won't require Log4j 1's AppenderSkeleton unless Log4j 1 is being used. It also eliminates the need to exclude the log4j 1.x and slf4j-log4j12 jars since they are now optional.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user rgoers opened a pull request:

          https://github.com/apache/hadoop/pull/299

          Support Log4j 2 and Logback

          This patch relates to https://issues.apache.org/jira/browse/HADOOP-12956. It makes the logging implementation in hadoop common optional and provides support for event counters in Log4j 2 and Logback in addition to Log4j 1.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/rgoers/hadoop trunk

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/299.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #299


          commit e0ed340172da1740da884e780250efc3e1b1519a
          Author: Ralph Goers <rgoers@apache.org>
          Date: 2017-11-26T06:57:03Z

          Support Log4j 2 and Logback


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user rgoers opened a pull request: https://github.com/apache/hadoop/pull/299 Support Log4j 2 and Logback This patch relates to https://issues.apache.org/jira/browse/HADOOP-12956 . It makes the logging implementation in hadoop common optional and provides support for event counters in Log4j 2 and Logback in addition to Log4j 1. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rgoers/hadoop trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/299.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #299 commit e0ed340172da1740da884e780250efc3e1b1519a Author: Ralph Goers <rgoers@apache.org> Date: 2017-11-26T06:57:03Z Support Log4j 2 and Logback
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Ralph:

          • we do want to move to Log4J 2.
          • As sean said, we've been wiating for as much aid in migration is possible...this JIRA is as good a place as any.
          • Why do we like to log4j.properties files? Ubiquity, and we understand them. All the Hadoop cluster management tools are adept at configuring them and pushing them out, which means that a migration has implications there too.
          • In test code we fiddle with logging settings, I don't know where this is done in production. It is done elsewhere (SPARK-14703).

          w.r.t SLF4J

          1. We've been slowly moving off commons logging to SLF4J for some years, long before worrying about the Log4J upgrades. Its API is an improvement of commons; even that move is unfinished but I look forward to the day we get to remove a dependency from everyone's classpath.
          2. We're still backporting a lot of code from 3.x to branch-2 and commercially supported variants thereof. Having a single unified log API which works everywhere makes cherry-picking much, much easier. We have enough pain backporting to want to add more.
          3. I like to give applications which use hadoop-the-libraries the choice of which back end to use. Commons logging delivered that, be it log4J or Apache Avalon, even as it added other bits of complexity (as indeed, SLF4J does).

          Do not read this as being any way critical of Log4j: we do intend to move to it as the back end, we do need to make progress on this. It's hard because of (a) the ubiquity of log statements, the ubiquity of log4j.property files, and the fact that log4j is a critical part of our infrastructure. If you haven't spend an afternoon with a 13MB log file from a single service in your IDE trying to work out WTF is wrong before eventually concluding that the problem was actually occurring in a different service and only surfacing locally, well, your weekends are different from ours. for those of us who do get to do that: we care about having the systems configure their logs such that we do at least get those 13 MB log files.

          Show
          stevel@apache.org Steve Loughran added a comment - Ralph: we do want to move to Log4J 2. As sean said, we've been wiating for as much aid in migration is possible...this JIRA is as good a place as any. Why do we like to log4j.properties files? Ubiquity, and we understand them. All the Hadoop cluster management tools are adept at configuring them and pushing them out, which means that a migration has implications there too. In test code we fiddle with logging settings, I don't know where this is done in production. It is done elsewhere ( SPARK-14703 ). w.r.t SLF4J We've been slowly moving off commons logging to SLF4J for some years, long before worrying about the Log4J upgrades. Its API is an improvement of commons; even that move is unfinished but I look forward to the day we get to remove a dependency from everyone's classpath. We're still backporting a lot of code from 3.x to branch-2 and commercially supported variants thereof. Having a single unified log API which works everywhere makes cherry-picking much, much easier. We have enough pain backporting to want to add more. I like to give applications which use hadoop-the-libraries the choice of which back end to use. Commons logging delivered that, be it log4J or Apache Avalon, even as it added other bits of complexity (as indeed, SLF4J does). Do not read this as being any way critical of Log4j: we do intend to move to it as the back end, we do need to make progress on this. It's hard because of (a) the ubiquity of log statements, the ubiquity of log4j.property files, and the fact that log4j is a critical part of our infrastructure. If you haven't spend an afternoon with a 13MB log file from a single service in your IDE trying to work out WTF is wrong before eventually concluding that the problem was actually occurring in a different service and only surfacing locally, well, your weekends are different from ours. for those of us who do get to do that: we care about having the systems configure their logs such that we do at least get those 13 MB log files.
          Hide
          busbey Sean Busbey added a comment -

          The only thing that supports the log4j 1 properties files is Log4j 1.x. That was declared EOL 2 years ago. The last release of Log4j 1 was 5 1/2 years ago. It doesn't run in Java 9 without hacking it.

          We're aware of the limitations of log4j 1. The burden on our operators for changing something as fundamental as logging is still something the project cares about. I'd be surprised if Hadoop took a hard look at Java 9 before late 2018.

          At some point you are going to have to get off of Log4j 1.

          The log4j team started an effort to create a properties file converter but it would only be able to convert Appenders & Layouts that are part of Log4j 1 itself. That is working to some degree but is still considered experimental. Any user created Appenders and Layouts would not be able to be migrated. As we would not be able to convert them to a Log4j 2 plugin.

          That said, we welcome any ideas or contributions anyone wants to contribute to make the migration easier.

          I get that it's frustrating to have folks not migrating. I'm a maintainer on a project that went through a major version change that didn't work well for operators (HBase in our 0.94 to 0.96 Event Horizon). The task was miserable for downstream folks as well as those on the project. That was just over 4 years ago and there are still folks running HBase 0.94.

          Frankly, it'd be very helpful for the Log4j community to state plainly and directly wether or not support for log4j 1 properties files will ever happen. We (the hadoop project as well as some other communities I watch) have gotten a mishmash of responses about it being in progress vs not feasible. A hard stance of "not happening" makes it easier for communities to plan their limited attention.

          I should also point out, SLF4J isn't really an answer for this problem either as Logback doesn't support Log4j 1 configurations and its migration tool can't handle custom Appenders or Layouts either.

          SLF4j is exactly the operational answer Hadoop needs. It lets us move our code's assumptions off of log4j1 while providing a log4j 1 bridge that will work with existing log4j 1 properties files. That way we can work incrementally on updating the code base while not requiring operators to change anything. Once we're done, operators who want to switch early can do so. As a project we can wait for our next major version to move the default to some other logging implementation.

          Show
          busbey Sean Busbey added a comment - The only thing that supports the log4j 1 properties files is Log4j 1.x. That was declared EOL 2 years ago. The last release of Log4j 1 was 5 1/2 years ago. It doesn't run in Java 9 without hacking it. We're aware of the limitations of log4j 1. The burden on our operators for changing something as fundamental as logging is still something the project cares about. I'd be surprised if Hadoop took a hard look at Java 9 before late 2018. At some point you are going to have to get off of Log4j 1. The log4j team started an effort to create a properties file converter but it would only be able to convert Appenders & Layouts that are part of Log4j 1 itself. That is working to some degree but is still considered experimental. Any user created Appenders and Layouts would not be able to be migrated. As we would not be able to convert them to a Log4j 2 plugin. That said, we welcome any ideas or contributions anyone wants to contribute to make the migration easier. I get that it's frustrating to have folks not migrating. I'm a maintainer on a project that went through a major version change that didn't work well for operators (HBase in our 0.94 to 0.96 Event Horizon). The task was miserable for downstream folks as well as those on the project. That was just over 4 years ago and there are still folks running HBase 0.94. Frankly, it'd be very helpful for the Log4j community to state plainly and directly wether or not support for log4j 1 properties files will ever happen. We (the hadoop project as well as some other communities I watch) have gotten a mishmash of responses about it being in progress vs not feasible. A hard stance of "not happening" makes it easier for communities to plan their limited attention. I should also point out, SLF4J isn't really an answer for this problem either as Logback doesn't support Log4j 1 configurations and its migration tool can't handle custom Appenders or Layouts either. SLF4j is exactly the operational answer Hadoop needs. It lets us move our code's assumptions off of log4j1 while providing a log4j 1 bridge that will work with existing log4j 1 properties files. That way we can work incrementally on updating the code base while not requiring operators to change anything. Once we're done, operators who want to switch early can do so. As a project we can wait for our next major version to move the default to some other logging implementation.
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment - - edited

          The only thing that supports the log4j 1 properties files is Log4j 1.x. That was declared EOL 2 years ago. The last release of Log4j 1 was 5 1/2 years ago. It doesn't run in Java 9 without hacking it.

          At some point you are going to have to get off of Log4j 1.

          The log4j team started an effort to create a properties file converter but it would only be able to convert Appenders & Layouts that are part of Log4j 1 itself. That is working to some degree but is still considered experimental. Any user created Appenders and Layouts would not be able to be migrated. As we would not be able to convert them to a Log4j 2 plugin.

          That said, we welcome any ideas or contributions anyone wants to contribute to make the migration easier.

          I should also point out, SLF4J isn't really an answer for this problem either as Logback doesn't support Log4j 1 configurations and its migration tool can't handle custom Appenders or Layouts either.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - - edited The only thing that supports the log4j 1 properties files is Log4j 1.x. That was declared EOL 2 years ago. The last release of Log4j 1 was 5 1/2 years ago. It doesn't run in Java 9 without hacking it. At some point you are going to have to get off of Log4j 1. The log4j team started an effort to create a properties file converter but it would only be able to convert Appenders & Layouts that are part of Log4j 1 itself. That is working to some degree but is still considered experimental. Any user created Appenders and Layouts would not be able to be migrated. As we would not be able to convert them to a Log4j 2 plugin. That said, we welcome any ideas or contributions anyone wants to contribute to make the migration easier. I should also point out, SLF4J isn't really an answer for this problem either as Logback doesn't support Log4j 1 configurations and its migration tool can't handle custom Appenders or Layouts either.
          Hide
          busbey Sean Busbey added a comment -

          We need to keep working with runtime deployments that rely on the log4j v1 properties files. To date that hasn't been possible with Log4j v2 as far as I know.

          Show
          busbey Sean Busbey added a comment - We need to keep working with runtime deployments that rely on the log4j v1 properties files. To date that hasn't been possible with Log4j v2 as far as I know.
          Hide
          ralph.goers@dslextreme.com Ralph Goers added a comment - - edited

          I have to question why you would move to SLF4J as the API when you can accomplish the same thing using the Log4j 2 API and the Log4j 2 API provides many more features than SLF4J.

          Show
          ralph.goers@dslextreme.com Ralph Goers added a comment - - edited I have to question why you would move to SLF4J as the API when you can accomplish the same thing using the Log4j 2 API and the Log4j 2 API provides many more features than SLF4J.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          2.9 will be out soon. Now I'm +1 for moving logging APIs over to slf4j in 3.0.

          Show
          ajisakaa Akira Ajisaka added a comment - 2.9 will be out soon. Now I'm +1 for moving logging APIs over to slf4j in 3.0.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Filed HADOOP-14289 for hadoop-common.

          Show
          ajisakaa Akira Ajisaka added a comment - Filed HADOOP-14289 for hadoop-common.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1 for moving logging APIs over to slf4j in 2.9.

          Show
          ajisakaa Akira Ajisaka added a comment - +1 for moving logging APIs over to slf4j in 2.9.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Happy for you to do the switch to SLF4J APIs; staying in log4j for our tests, properties &c something I'll expect to continue though. That shouldn't impact what you do downstream with the hadoop JARs

          Show
          stevel@apache.org Steve Loughran added a comment - Happy for you to do the switch to SLF4J APIs; staying in log4j for our tests, properties &c something I'll expect to continue though. That shouldn't impact what you do downstream with the hadoop JARs
          Hide
          wheat9 Haohui Mai added a comment -

          In our production environment we saw various bottlenecks in NN due to log4j, even with asynchronous logging turned on.

          Having the option of moving towards log4j2 via slf4j, even it means that it is backward-incompatible is still hugely beneficial.

          I think it is still possible to pull it off although it will touch a lot of code:

          $ find . -name "*.java"|xargs grep "log4j\."|grep "import" -c
          289
          $ find . -name "*.java"|xargs grep "log4j\."|grep "import"|grep "/test/" -c
          242
          
          Show
          wheat9 Haohui Mai added a comment - In our production environment we saw various bottlenecks in NN due to log4j, even with asynchronous logging turned on. Having the option of moving towards log4j2 via slf4j, even it means that it is backward-incompatible is still hugely beneficial. I think it is still possible to pull it off although it will touch a lot of code: $ find . -name "*.java"|xargs grep "log4j\."|grep "import" -c 289 $ find . -name "*.java"|xargs grep "log4j\."|grep "import"|grep "/test/" -c 242
          Hide
          andrew.wang Andrew Wang added a comment -

          I'm fine with moving logging APIs over to slf4j in 2.x, as Steve described above.

          2.8 is trying to wrap a release right now, so it'd be safer to target this for 2.9 since this is going to touch a lot of code.

          Show
          andrew.wang Andrew Wang added a comment - I'm fine with moving logging APIs over to slf4j in 2.x, as Steve described above. 2.8 is trying to wrap a release right now, so it'd be safer to target this for 2.9 since this is going to touch a lot of code.
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          Unfortunately, we did not manage to finalize the support for Log4j 1 configuration files in Log4j 2.7. There is some experimental support in 2.7, but not complete or stable yet. We will continue to work on this, but I cannot promise any particular release or time right now.

          However, this should not block you from making progress in Hadoop. If you find anything blocking you, please raise the issue as a comment here and we will have a look.

          Show
          mikaelstaldal Mikael Ståldal added a comment - Unfortunately, we did not manage to finalize the support for Log4j 1 configuration files in Log4j 2.7. There is some experimental support in 2.7, but not complete or stable yet. We will continue to work on this, but I cannot promise any particular release or time right now. However, this should not block you from making progress in Hadoop. If you find anything blocking you, please raise the issue as a comment here and we will have a look.
          Hide
          roniburd Roni Burd added a comment -

          Sounds good. Just to be on the same page: you guys are friendly to the proposal to bake this into 2.8 in addition to trunk?

          Show
          roniburd Roni Burd added a comment - Sounds good. Just to be on the same page: you guys are friendly to the proposal to bake this into 2.8 in addition to trunk?
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Certainly moving to the slf4j APIs is something to do. The simple approach: just change the type of the log used. The better one: at least for info and above levels, just rely on slf4 varargs expansion.

          if you look at how log4j is explicitly used in the code, its in some of the tests, where its log is cranked back. Something will need to be done for log4j 2 there

          Show
          stevel@apache.org Steve Loughran added a comment - Certainly moving to the slf4j APIs is something to do. The simple approach: just change the type of the log used. The better one: at least for info and above levels, just rely on slf4 varargs expansion. if you look at how log4j is explicitly used in the code, its in some of the tests, where its log is cranked back. Something will need to be done for log4j 2 there
          Hide
          roniburd Roni Burd added a comment - - edited

          Hi, I'm a ppal dev in MSFT and my team owns a massive YARN deployment. I was profiling our latest production push and the log4j library takes a big chunk of processing time to the point of even blocking other allocate() calls in YARM RM. (happy to send some traces if anyone is interested)

          I was reading about the challenges in upgrading to log4j2 and how we are waiting for log4j2 2.7 for backward compat on the properties file. While the log4j guys are working on that, was wondering if there is anything preventing us from making forward progress in parallel? Also, I can see how this change could give a perf boost to older branches like 2.8 when running at scale without major risk to destabilize functionality.

          I'm more than happy to help with patches as we might need this change for our scale anyways and because we dont have strict compat issues that prevents us from moving now. Was really looking for some guidance in order to contribute back (our team does regular contributions so we know the process - but not on this specific area nor me personally)

          Thanks!

          Show
          roniburd Roni Burd added a comment - - edited Hi, I'm a ppal dev in MSFT and my team owns a massive YARN deployment. I was profiling our latest production push and the log4j library takes a big chunk of processing time to the point of even blocking other allocate() calls in YARM RM. (happy to send some traces if anyone is interested) I was reading about the challenges in upgrading to log4j2 and how we are waiting for log4j2 2.7 for backward compat on the properties file. While the log4j guys are working on that, was wondering if there is anything preventing us from making forward progress in parallel? Also, I can see how this change could give a perf boost to older branches like 2.8 when running at scale without major risk to destabilize functionality. I'm more than happy to help with patches as we might need this change for our scale anyways and because we dont have strict compat issues that prevents us from moving now. Was really looking for some guidance in order to contribute back (our team does regular contributions so we know the process - but not on this specific area nor me personally) Thanks!
          Hide
          cnauroth Chris Nauroth added a comment -

          Mikael Ståldal, backward-compatibility with the Log4J 1 properties file format would be hugely beneficial for us. Thank you. I'm linking to LOG4J2-63, which seems to be the master JIRA tracking that effort against Log4J 2.7.

          Show
          cnauroth Chris Nauroth added a comment - Mikael Ståldal , backward-compatibility with the Log4J 1 properties file format would be hugely beneficial for us. Thank you. I'm linking to LOG4J2-63 , which seems to be the master JIRA tracking that effort against Log4J 2.7.
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          Yes, we are working on that right now. The plan is to have that ready in the next release of Log4j (2.7).

          Show
          mikaelstaldal Mikael Ståldal added a comment - Yes, we are working on that right now. The plan is to have that ready in the next release of Log4j (2.7).
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This is going to be fun. How about starting at discussion on common-dev &c?

          some parts of the code (hdfs-client) are now log-backend neutral. Assuming all uses of log4j APIs are in tests, maybe some wrapper could isolate the conversion, or provide a flag to indicate if level tuning was allowed; test cases could be skipped if not.

          someone could then do an experimental branch using logback to see what happens. It could also allow people to jump to log4j2 sooner rather than later, if they so choose.

          Show
          stevel@apache.org Steve Loughran added a comment - This is going to be fun. How about starting at discussion on common-dev &c? some parts of the code (hdfs-client) are now log-backend neutral. Assuming all uses of log4j APIs are in tests, maybe some wrapper could isolate the conversion, or provide a flag to indicate if level tuning was allowed; test cases could be skipped if not. someone could then do an experimental branch using logback to see what happens. It could also allow people to jump to log4j2 sooner rather than later, if they so choose.
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          Log4j 2 can handle logging from commons-logging: http://logging.apache.org/log4j/2.x/log4j-jcl/index.html

          Show
          mikaelstaldal Mikael Ståldal added a comment - Log4j 2 can handle logging from commons-logging: http://logging.apache.org/log4j/2.x/log4j-jcl/index.html
          Hide
          mikaelstaldal Mikael Ståldal added a comment - - edited

          Mixing two different logging implementations like Log4j 1 and Log4j 2 is not recommended and will most likely lead to problems. What you can do is to use the Log4j 1 bridge for Log4j 2: http://logging.apache.org/log4j/2.x/log4j-1.2-api/index.html

          Show
          mikaelstaldal Mikael Ståldal added a comment - - edited Mixing two different logging implementations like Log4j 1 and Log4j 2 is not recommended and will most likely lead to problems. What you can do is to use the Log4j 1 bridge for Log4j 2: http://logging.apache.org/log4j/2.x/log4j-1.2-api/index.html
          Hide
          aw Allen Wittenauer added a comment -

          Again, to re-iterate:

          If we have to make such a drastically incompatible change that's going to break all of our users' configuration files, then we should probably investigate whether we want to stay on log4j or move to something else (logback?).

          Show
          aw Allen Wittenauer added a comment - Again, to re-iterate: If we have to make such a drastically incompatible change that's going to break all of our users' configuration files, then we should probably investigate whether we want to stay on log4j or move to something else (logback?).
          Hide
          busbey Sean Busbey added a comment -

          We suggest upgrading to Log4j 2.x, we can offer help to do so.

          Any chance of adding the ability to parse log4j 1.2.z properties files? Or a migration tool?

          Show
          busbey Sean Busbey added a comment - We suggest upgrading to Log4j 2.x, we can offer help to do so. Any chance of adding the ability to parse log4j 1.2.z properties files? Or a migration tool?
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          Hi,

          I am member of Apache Logging project PMC.

          Due to various issues in Log4j 1.2.x, it will no longer work in JDK9: http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008654.html

          As Log4j 1.x is EOL, these issues will not be fixed. We suggest upgrading to Log4j 2.x, we can offer help to do so.

          Show
          mikaelstaldal Mikael Ståldal added a comment - Hi, I am member of Apache Logging project PMC. Due to various issues in Log4j 1.2.x, it will no longer work in JDK9: http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008654.html As Log4j 1.x is EOL, these issues will not be fixed. We suggest upgrading to Log4j 2.x, we can offer help to do so.
          Hide
          aw Allen Wittenauer added a comment -

          Is it time to completely dump log4j and move to something else entirely?

          Show
          aw Allen Wittenauer added a comment - Is it time to completely dump log4j and move to something else entirely?
          Hide
          cnauroth Chris Nauroth added a comment -

          Has anyone spoken to the Log4J team about this?

          From what I can tell, the Log4J 2 domain model has changed so significantly that they don't consider it workable to attempt full backwards-compatibility with Log4J 1 style configuration. We could certainly ask again though.

          I did have a brief conversation about losing our ability to switch logging levels by passing -D arguments. They had a suggestion for plugging in to the Log4J 2 internals to accomplish this, but I haven't had time to try it.

          http://mail-archives.apache.org/mod_mbox/logging-log4j-user/201602.mbox/%3CD35E0663-D78B-4DAF-A26A-F5A27E8A14FC%40dslextreme.com%3E

          I don't have any concrete information about the mixing problem at this point. I expect it will require experimentation to figure out what is possible.

          Would the log tuning servlets work? They are invaluable for debugging live services.

          This is implemented with Commons Logging calls to change the levels. I didn't investigate anything related to Commons Logging, because ZooKeeper has already fully migrated to SLF4J as the logging API. Even if this doesn't work, I'd expect it could be made to work by switching the code to call equivalent Log4J 2 APIs.

          we can still continue the move to SLF4J APIs, even for branch-2 code.

          +1

          Show
          cnauroth Chris Nauroth added a comment - Has anyone spoken to the Log4J team about this? From what I can tell, the Log4J 2 domain model has changed so significantly that they don't consider it workable to attempt full backwards-compatibility with Log4J 1 style configuration. We could certainly ask again though. I did have a brief conversation about losing our ability to switch logging levels by passing -D arguments. They had a suggestion for plugging in to the Log4J 2 internals to accomplish this, but I haven't had time to try it. http://mail-archives.apache.org/mod_mbox/logging-log4j-user/201602.mbox/%3CD35E0663-D78B-4DAF-A26A-F5A27E8A14FC%40dslextreme.com%3E I don't have any concrete information about the mixing problem at this point. I expect it will require experimentation to figure out what is possible. Would the log tuning servlets work? They are invaluable for debugging live services. This is implemented with Commons Logging calls to change the levels. I didn't investigate anything related to Commons Logging, because ZooKeeper has already fully migrated to SLF4J as the logging API. Even if this doesn't work, I'd expect it could be made to work by switching the code to call equivalent Log4J 2 APIs. we can still continue the move to SLF4J APIs, even for branch-2 code. +1
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Has anyone spoken to the Log4J team about this?

          Chris —that's an interesting q. about mixing. We'd need to make sure the SLF4J to Log4j2 JAR was on the CP, and that log4j was off it. Would commons-logging be able to feed in to log4j2 then? Or is a full move off it required? As that's more expensive, especially as there are so many nested libraries which use what has been the defacto Java logging API since the Avalon framework was written.

          Would the log tuning servlets work? They are invaluable for debugging live services.

          we can still continue the move to SLF4J APIs, even for branch-2 code. it works great as an API...it's the new back end which is the troublespot

          Show
          stevel@apache.org Steve Loughran added a comment - Has anyone spoken to the Log4J team about this? Chris —that's an interesting q. about mixing. We'd need to make sure the SLF4J to Log4j2 JAR was on the CP, and that log4j was off it. Would commons-logging be able to feed in to log4j2 then? Or is a full move off it required? As that's more expensive, especially as there are so many nested libraries which use what has been the defacto Java logging API since the Avalon framework was written. Would the log tuning servlets work? They are invaluable for debugging live services. we can still continue the move to SLF4J APIs, even for branch-2 code. it works great as an API...it's the new back end which is the troublespot
          Hide
          apurtell Andrew Purtell added a comment -

          We also evaluated a move to log4j2 for HBase on HBASE-10092, and likewise came to the conclusion it is impossible to achieve backwards compatibility for operators and tools. A change like this can't go in on a minor release. (At least, HBase cannot do that, according to our published version compatibility guidelines.) The killer issue is no backwards compatibility for the properties based configuration files. Operators, not unreasonably, expect a minor or patch version increment should continue to "just work" with existing configuration files and tooling, not completely break all logging.

          I see after LOG4J2-952 there is a "PropertiesConfigurationFactory", so a properties based configuration file format is again possible, but this isn't helpful and probably not desired because you get the limitations of a properties file format without v1 compatibility.

          Show
          apurtell Andrew Purtell added a comment - We also evaluated a move to log4j2 for HBase on HBASE-10092 , and likewise came to the conclusion it is impossible to achieve backwards compatibility for operators and tools. A change like this can't go in on a minor release. (At least, HBase cannot do that, according to our published version compatibility guidelines.) The killer issue is no backwards compatibility for the properties based configuration files. Operators, not unreasonably, expect a minor or patch version increment should continue to "just work" with existing configuration files and tooling, not completely break all logging. I see after LOG4J2-952 there is a "PropertiesConfigurationFactory", so a properties based configuration file format is again possible, but this isn't helpful and probably not desired because you get the limitations of a properties file format without v1 compatibility.
          Hide
          cnauroth Chris Nauroth added a comment -

          A few more notes from my stab at this in ZooKeeper:

          1. The actual code changes for the logging are the trivial part, even if somewhat time-consuming. ZooKeeper had already completed an almost full migration to the SLF4J API, so that helped.
          2. Tests that rely on log capture and inspection of messages needed changes to hook into the new Log4J 2 classes.
          3. There is no option for backwards-compatibility with the Log4J 1 configuration format.
          4. I couldn't figure out a way to preserve our trick of passing JVM arguments to tweak the log level, i.e. -Dhadoop.security.logger=INFO,DRFAS. The Log4J 2 configuration format is much more complex, so there isn't a simple place to do a token replacement of the "INFO,DRFAS". From a brief chat on the Log4J user list, that community had some ideas about coding some Log4J 2 customizations that migh help me achieve that, but I didn't get a chance to explore that.
          5. What happens when something tries to use a mix of Log4J 1 and Log4J 2? Does it bomb? Does it work, but split the messages between different files controlled by the 2 different frameworks? Does this need to be handled as a massive full ecosystem upgrade, a la Protobuf?
          Show
          cnauroth Chris Nauroth added a comment - A few more notes from my stab at this in ZooKeeper: The actual code changes for the logging are the trivial part, even if somewhat time-consuming. ZooKeeper had already completed an almost full migration to the SLF4J API, so that helped. Tests that rely on log capture and inspection of messages needed changes to hook into the new Log4J 2 classes. There is no option for backwards-compatibility with the Log4J 1 configuration format. I couldn't figure out a way to preserve our trick of passing JVM arguments to tweak the log level, i.e. -Dhadoop.security.logger=INFO,DRFAS . The Log4J 2 configuration format is much more complex, so there isn't a simple place to do a token replacement of the "INFO,DRFAS". From a brief chat on the Log4J user list, that community had some ideas about coding some Log4J 2 customizations that migh help me achieve that, but I didn't get a chance to explore that. What happens when something tries to use a mix of Log4J 1 and Log4J 2? Does it bomb? Does it work, but split the messages between different files controlled by the 2 different frameworks? Does this need to be handled as a massive full ecosystem upgrade, a la Protobuf?
          Hide
          gopalv Gopal V added a comment -

          if commons-logging supports log4j2.x

          LLAP moved DFSClient etc logging to log4j2 via the jcl bridge - http://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-jcl

          The thing that broke however was the .properties syntax & the /logs dynamic configuration servlet, which assumes log4j1.x

          Show
          gopalv Gopal V added a comment - if commons-logging supports log4j2.x LLAP moved DFSClient etc logging to log4j2 via the jcl bridge - http://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-jcl The thing that broke however was the .properties syntax & the /logs dynamic configuration servlet, which assumes log4j1.x
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This may be unrelated with a move to SLF4J, if commons-logging supports log4j2.x. We could do that migration today on a package by package basis, with the low-cost approach just being to change the type of the logger, without going through the log statements and redoing them.

          Back end wise, lets look at your ZK move first

          Show
          stevel@apache.org Steve Loughran added a comment - This may be unrelated with a move to SLF4J, if commons-logging supports log4j2.x. We could do that migration today on a package by package basis, with the low-cost approach just being to change the type of the logger, without going through the log statements and redoing them. Back end wise, lets look at your ZK move first
          Hide
          cnauroth Chris Nauroth added a comment -

          I recently explored a Log4J 2 migration for ZooKeeper. My notes and a work-in-progress patch are on ZOOKEEPER-2342. We came to the conclusion that it's impossible to achieve backwards-compatibility for operators and tools that edit Log4J configuration, so I've put the work on hold at this point.

          I agree with the use of "inevitable" in the JIRA title, but it's going to be a very difficult change, likely requiring ecosystem coordination.

          Show
          cnauroth Chris Nauroth added a comment - I recently explored a Log4J 2 migration for ZooKeeper. My notes and a work-in-progress patch are on ZOOKEEPER-2342 . We came to the conclusion that it's impossible to achieve backwards-compatibility for operators and tools that edit Log4J configuration, so I've put the work on hold at this point. I agree with the use of "inevitable" in the JIRA title, but it's going to be a very difficult change, likely requiring ecosystem coordination.

            People

            • Assignee:
              wheat9 Haohui Mai
              Reporter:
              gopalv Gopal V
            • Votes:
              0 Vote for this issue
              Watchers:
              38 Start watching this issue

              Dates

              • Created:
                Updated:

                Development