Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      5 August 2015 --The Apache Logging Services™ Project Management Committee (PMC) has announced that the Log4j™ 1.x logging framework has reached its end of life (EOL) and is no longer officially supported.

      https://blogs.apache.org/foundation/entry/apache_logging_services_project_announces

      A whole framework log4j2 upgrade has to be synchronized, partly for improved performance brought about by log4j2.

      https://logging.apache.org/log4j/2.x/manual/async.html#Performance

        Issue Links

          Activity

          Hide
          cnauroth Chris Nauroth added a comment -

          I recently explored a Log4J 2 migration for ZooKeeper. My notes and a work-in-progress patch are on ZOOKEEPER-2342. We came to the conclusion that it's impossible to achieve backwards-compatibility for operators and tools that edit Log4J configuration, so I've put the work on hold at this point.

          I agree with the use of "inevitable" in the JIRA title, but it's going to be a very difficult change, likely requiring ecosystem coordination.

          Show
          cnauroth Chris Nauroth added a comment - I recently explored a Log4J 2 migration for ZooKeeper. My notes and a work-in-progress patch are on ZOOKEEPER-2342 . We came to the conclusion that it's impossible to achieve backwards-compatibility for operators and tools that edit Log4J configuration, so I've put the work on hold at this point. I agree with the use of "inevitable" in the JIRA title, but it's going to be a very difficult change, likely requiring ecosystem coordination.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This may be unrelated with a move to SLF4J, if commons-logging supports log4j2.x. We could do that migration today on a package by package basis, with the low-cost approach just being to change the type of the logger, without going through the log statements and redoing them.

          Back end wise, lets look at your ZK move first

          Show
          stevel@apache.org Steve Loughran added a comment - This may be unrelated with a move to SLF4J, if commons-logging supports log4j2.x. We could do that migration today on a package by package basis, with the low-cost approach just being to change the type of the logger, without going through the log statements and redoing them. Back end wise, lets look at your ZK move first
          Hide
          gopalv Gopal V added a comment -

          if commons-logging supports log4j2.x

          LLAP moved DFSClient etc logging to log4j2 via the jcl bridge - http://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-jcl

          The thing that broke however was the .properties syntax & the /logs dynamic configuration servlet, which assumes log4j1.x

          Show
          gopalv Gopal V added a comment - if commons-logging supports log4j2.x LLAP moved DFSClient etc logging to log4j2 via the jcl bridge - http://mvnrepository.com/artifact/org.apache.logging.log4j/log4j-jcl The thing that broke however was the .properties syntax & the /logs dynamic configuration servlet, which assumes log4j1.x
          Hide
          cnauroth Chris Nauroth added a comment -

          A few more notes from my stab at this in ZooKeeper:

          1. The actual code changes for the logging are the trivial part, even if somewhat time-consuming. ZooKeeper had already completed an almost full migration to the SLF4J API, so that helped.
          2. Tests that rely on log capture and inspection of messages needed changes to hook into the new Log4J 2 classes.
          3. There is no option for backwards-compatibility with the Log4J 1 configuration format.
          4. I couldn't figure out a way to preserve our trick of passing JVM arguments to tweak the log level, i.e. -Dhadoop.security.logger=INFO,DRFAS. The Log4J 2 configuration format is much more complex, so there isn't a simple place to do a token replacement of the "INFO,DRFAS". From a brief chat on the Log4J user list, that community had some ideas about coding some Log4J 2 customizations that migh help me achieve that, but I didn't get a chance to explore that.
          5. What happens when something tries to use a mix of Log4J 1 and Log4J 2? Does it bomb? Does it work, but split the messages between different files controlled by the 2 different frameworks? Does this need to be handled as a massive full ecosystem upgrade, a la Protobuf?
          Show
          cnauroth Chris Nauroth added a comment - A few more notes from my stab at this in ZooKeeper: The actual code changes for the logging are the trivial part, even if somewhat time-consuming. ZooKeeper had already completed an almost full migration to the SLF4J API, so that helped. Tests that rely on log capture and inspection of messages needed changes to hook into the new Log4J 2 classes. There is no option for backwards-compatibility with the Log4J 1 configuration format. I couldn't figure out a way to preserve our trick of passing JVM arguments to tweak the log level, i.e. -Dhadoop.security.logger=INFO,DRFAS . The Log4J 2 configuration format is much more complex, so there isn't a simple place to do a token replacement of the "INFO,DRFAS". From a brief chat on the Log4J user list, that community had some ideas about coding some Log4J 2 customizations that migh help me achieve that, but I didn't get a chance to explore that. What happens when something tries to use a mix of Log4J 1 and Log4J 2? Does it bomb? Does it work, but split the messages between different files controlled by the 2 different frameworks? Does this need to be handled as a massive full ecosystem upgrade, a la Protobuf?
          Hide
          apurtell Andrew Purtell added a comment -

          We also evaluated a move to log4j2 for HBase on HBASE-10092, and likewise came to the conclusion it is impossible to achieve backwards compatibility for operators and tools. A change like this can't go in on a minor release. (At least, HBase cannot do that, according to our published version compatibility guidelines.) The killer issue is no backwards compatibility for the properties based configuration files. Operators, not unreasonably, expect a minor or patch version increment should continue to "just work" with existing configuration files and tooling, not completely break all logging.

          I see after LOG4J2-952 there is a "PropertiesConfigurationFactory", so a properties based configuration file format is again possible, but this isn't helpful and probably not desired because you get the limitations of a properties file format without v1 compatibility.

          Show
          apurtell Andrew Purtell added a comment - We also evaluated a move to log4j2 for HBase on HBASE-10092 , and likewise came to the conclusion it is impossible to achieve backwards compatibility for operators and tools. A change like this can't go in on a minor release. (At least, HBase cannot do that, according to our published version compatibility guidelines.) The killer issue is no backwards compatibility for the properties based configuration files. Operators, not unreasonably, expect a minor or patch version increment should continue to "just work" with existing configuration files and tooling, not completely break all logging. I see after LOG4J2-952 there is a "PropertiesConfigurationFactory", so a properties based configuration file format is again possible, but this isn't helpful and probably not desired because you get the limitations of a properties file format without v1 compatibility.
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Has anyone spoken to the Log4J team about this?

          Chris —that's an interesting q. about mixing. We'd need to make sure the SLF4J to Log4j2 JAR was on the CP, and that log4j was off it. Would commons-logging be able to feed in to log4j2 then? Or is a full move off it required? As that's more expensive, especially as there are so many nested libraries which use what has been the defacto Java logging API since the Avalon framework was written.

          Would the log tuning servlets work? They are invaluable for debugging live services.

          we can still continue the move to SLF4J APIs, even for branch-2 code. it works great as an API...it's the new back end which is the troublespot

          Show
          stevel@apache.org Steve Loughran added a comment - Has anyone spoken to the Log4J team about this? Chris —that's an interesting q. about mixing. We'd need to make sure the SLF4J to Log4j2 JAR was on the CP, and that log4j was off it. Would commons-logging be able to feed in to log4j2 then? Or is a full move off it required? As that's more expensive, especially as there are so many nested libraries which use what has been the defacto Java logging API since the Avalon framework was written. Would the log tuning servlets work? They are invaluable for debugging live services. we can still continue the move to SLF4J APIs, even for branch-2 code. it works great as an API...it's the new back end which is the troublespot
          Hide
          cnauroth Chris Nauroth added a comment -

          Has anyone spoken to the Log4J team about this?

          From what I can tell, the Log4J 2 domain model has changed so significantly that they don't consider it workable to attempt full backwards-compatibility with Log4J 1 style configuration. We could certainly ask again though.

          I did have a brief conversation about losing our ability to switch logging levels by passing -D arguments. They had a suggestion for plugging in to the Log4J 2 internals to accomplish this, but I haven't had time to try it.

          http://mail-archives.apache.org/mod_mbox/logging-log4j-user/201602.mbox/%3CD35E0663-D78B-4DAF-A26A-F5A27E8A14FC%40dslextreme.com%3E

          I don't have any concrete information about the mixing problem at this point. I expect it will require experimentation to figure out what is possible.

          Would the log tuning servlets work? They are invaluable for debugging live services.

          This is implemented with Commons Logging calls to change the levels. I didn't investigate anything related to Commons Logging, because ZooKeeper has already fully migrated to SLF4J as the logging API. Even if this doesn't work, I'd expect it could be made to work by switching the code to call equivalent Log4J 2 APIs.

          we can still continue the move to SLF4J APIs, even for branch-2 code.

          +1

          Show
          cnauroth Chris Nauroth added a comment - Has anyone spoken to the Log4J team about this? From what I can tell, the Log4J 2 domain model has changed so significantly that they don't consider it workable to attempt full backwards-compatibility with Log4J 1 style configuration. We could certainly ask again though. I did have a brief conversation about losing our ability to switch logging levels by passing -D arguments. They had a suggestion for plugging in to the Log4J 2 internals to accomplish this, but I haven't had time to try it. http://mail-archives.apache.org/mod_mbox/logging-log4j-user/201602.mbox/%3CD35E0663-D78B-4DAF-A26A-F5A27E8A14FC%40dslextreme.com%3E I don't have any concrete information about the mixing problem at this point. I expect it will require experimentation to figure out what is possible. Would the log tuning servlets work? They are invaluable for debugging live services. This is implemented with Commons Logging calls to change the levels. I didn't investigate anything related to Commons Logging, because ZooKeeper has already fully migrated to SLF4J as the logging API. Even if this doesn't work, I'd expect it could be made to work by switching the code to call equivalent Log4J 2 APIs. we can still continue the move to SLF4J APIs, even for branch-2 code. +1
          Hide
          aw Allen Wittenauer added a comment -

          Is it time to completely dump log4j and move to something else entirely?

          Show
          aw Allen Wittenauer added a comment - Is it time to completely dump log4j and move to something else entirely?
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          Hi,

          I am member of Apache Logging project PMC.

          Due to various issues in Log4j 1.2.x, it will no longer work in JDK9: http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008654.html

          As Log4j 1.x is EOL, these issues will not be fixed. We suggest upgrading to Log4j 2.x, we can offer help to do so.

          Show
          mikaelstaldal Mikael Ståldal added a comment - Hi, I am member of Apache Logging project PMC. Due to various issues in Log4j 1.2.x, it will no longer work in JDK9: http://mail.openjdk.java.net/pipermail/jigsaw-dev/2016-July/008654.html As Log4j 1.x is EOL, these issues will not be fixed. We suggest upgrading to Log4j 2.x, we can offer help to do so.
          Hide
          busbey Sean Busbey added a comment -

          We suggest upgrading to Log4j 2.x, we can offer help to do so.

          Any chance of adding the ability to parse log4j 1.2.z properties files? Or a migration tool?

          Show
          busbey Sean Busbey added a comment - We suggest upgrading to Log4j 2.x, we can offer help to do so. Any chance of adding the ability to parse log4j 1.2.z properties files? Or a migration tool?
          Hide
          aw Allen Wittenauer added a comment -

          Again, to re-iterate:

          If we have to make such a drastically incompatible change that's going to break all of our users' configuration files, then we should probably investigate whether we want to stay on log4j or move to something else (logback?).

          Show
          aw Allen Wittenauer added a comment - Again, to re-iterate: If we have to make such a drastically incompatible change that's going to break all of our users' configuration files, then we should probably investigate whether we want to stay on log4j or move to something else (logback?).
          Hide
          mikaelstaldal Mikael Ståldal added a comment - - edited

          Mixing two different logging implementations like Log4j 1 and Log4j 2 is not recommended and will most likely lead to problems. What you can do is to use the Log4j 1 bridge for Log4j 2: http://logging.apache.org/log4j/2.x/log4j-1.2-api/index.html

          Show
          mikaelstaldal Mikael Ståldal added a comment - - edited Mixing two different logging implementations like Log4j 1 and Log4j 2 is not recommended and will most likely lead to problems. What you can do is to use the Log4j 1 bridge for Log4j 2: http://logging.apache.org/log4j/2.x/log4j-1.2-api/index.html
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          Log4j 2 can handle logging from commons-logging: http://logging.apache.org/log4j/2.x/log4j-jcl/index.html

          Show
          mikaelstaldal Mikael Ståldal added a comment - Log4j 2 can handle logging from commons-logging: http://logging.apache.org/log4j/2.x/log4j-jcl/index.html
          Hide
          stevel@apache.org Steve Loughran added a comment -

          This is going to be fun. How about starting at discussion on common-dev &c?

          some parts of the code (hdfs-client) are now log-backend neutral. Assuming all uses of log4j APIs are in tests, maybe some wrapper could isolate the conversion, or provide a flag to indicate if level tuning was allowed; test cases could be skipped if not.

          someone could then do an experimental branch using logback to see what happens. It could also allow people to jump to log4j2 sooner rather than later, if they so choose.

          Show
          stevel@apache.org Steve Loughran added a comment - This is going to be fun. How about starting at discussion on common-dev &c? some parts of the code (hdfs-client) are now log-backend neutral. Assuming all uses of log4j APIs are in tests, maybe some wrapper could isolate the conversion, or provide a flag to indicate if level tuning was allowed; test cases could be skipped if not. someone could then do an experimental branch using logback to see what happens. It could also allow people to jump to log4j2 sooner rather than later, if they so choose.
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          Yes, we are working on that right now. The plan is to have that ready in the next release of Log4j (2.7).

          Show
          mikaelstaldal Mikael Ståldal added a comment - Yes, we are working on that right now. The plan is to have that ready in the next release of Log4j (2.7).
          Hide
          cnauroth Chris Nauroth added a comment -

          Mikael Ståldal, backward-compatibility with the Log4J 1 properties file format would be hugely beneficial for us. Thank you. I'm linking to LOG4J2-63, which seems to be the master JIRA tracking that effort against Log4J 2.7.

          Show
          cnauroth Chris Nauroth added a comment - Mikael Ståldal , backward-compatibility with the Log4J 1 properties file format would be hugely beneficial for us. Thank you. I'm linking to LOG4J2-63 , which seems to be the master JIRA tracking that effort against Log4J 2.7.
          Hide
          roniburd Roni Burd added a comment - - edited

          Hi, I'm a ppal dev in MSFT and my team owns a massive YARN deployment. I was profiling our latest production push and the log4j library takes a big chunk of processing time to the point of even blocking other allocate() calls in YARM RM. (happy to send some traces if anyone is interested)

          I was reading about the challenges in upgrading to log4j2 and how we are waiting for log4j2 2.7 for backward compat on the properties file. While the log4j guys are working on that, was wondering if there is anything preventing us from making forward progress in parallel? Also, I can see how this change could give a perf boost to older branches like 2.8 when running at scale without major risk to destabilize functionality.

          I'm more than happy to help with patches as we might need this change for our scale anyways and because we dont have strict compat issues that prevents us from moving now. Was really looking for some guidance in order to contribute back (our team does regular contributions so we know the process - but not on this specific area nor me personally)

          Thanks!

          Show
          roniburd Roni Burd added a comment - - edited Hi, I'm a ppal dev in MSFT and my team owns a massive YARN deployment. I was profiling our latest production push and the log4j library takes a big chunk of processing time to the point of even blocking other allocate() calls in YARM RM. (happy to send some traces if anyone is interested) I was reading about the challenges in upgrading to log4j2 and how we are waiting for log4j2 2.7 for backward compat on the properties file. While the log4j guys are working on that, was wondering if there is anything preventing us from making forward progress in parallel? Also, I can see how this change could give a perf boost to older branches like 2.8 when running at scale without major risk to destabilize functionality. I'm more than happy to help with patches as we might need this change for our scale anyways and because we dont have strict compat issues that prevents us from moving now. Was really looking for some guidance in order to contribute back (our team does regular contributions so we know the process - but not on this specific area nor me personally) Thanks!
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Certainly moving to the slf4j APIs is something to do. The simple approach: just change the type of the log used. The better one: at least for info and above levels, just rely on slf4 varargs expansion.

          if you look at how log4j is explicitly used in the code, its in some of the tests, where its log is cranked back. Something will need to be done for log4j 2 there

          Show
          stevel@apache.org Steve Loughran added a comment - Certainly moving to the slf4j APIs is something to do. The simple approach: just change the type of the log used. The better one: at least for info and above levels, just rely on slf4 varargs expansion. if you look at how log4j is explicitly used in the code, its in some of the tests, where its log is cranked back. Something will need to be done for log4j 2 there
          Hide
          roniburd Roni Burd added a comment -

          Sounds good. Just to be on the same page: you guys are friendly to the proposal to bake this into 2.8 in addition to trunk?

          Show
          roniburd Roni Burd added a comment - Sounds good. Just to be on the same page: you guys are friendly to the proposal to bake this into 2.8 in addition to trunk?
          Hide
          mikaelstaldal Mikael Ståldal added a comment -

          Unfortunately, we did not manage to finalize the support for Log4j 1 configuration files in Log4j 2.7. There is some experimental support in 2.7, but not complete or stable yet. We will continue to work on this, but I cannot promise any particular release or time right now.

          However, this should not block you from making progress in Hadoop. If you find anything blocking you, please raise the issue as a comment here and we will have a look.

          Show
          mikaelstaldal Mikael Ståldal added a comment - Unfortunately, we did not manage to finalize the support for Log4j 1 configuration files in Log4j 2.7. There is some experimental support in 2.7, but not complete or stable yet. We will continue to work on this, but I cannot promise any particular release or time right now. However, this should not block you from making progress in Hadoop. If you find anything blocking you, please raise the issue as a comment here and we will have a look.
          Hide
          andrew.wang Andrew Wang added a comment -

          I'm fine with moving logging APIs over to slf4j in 2.x, as Steve described above.

          2.8 is trying to wrap a release right now, so it'd be safer to target this for 2.9 since this is going to touch a lot of code.

          Show
          andrew.wang Andrew Wang added a comment - I'm fine with moving logging APIs over to slf4j in 2.x, as Steve described above. 2.8 is trying to wrap a release right now, so it'd be safer to target this for 2.9 since this is going to touch a lot of code.
          Hide
          wheat9 Haohui Mai added a comment -

          In our production environment we saw various bottlenecks in NN due to log4j, even with asynchronous logging turned on.

          Having the option of moving towards log4j2 via slf4j, even it means that it is backward-incompatible is still hugely beneficial.

          I think it is still possible to pull it off although it will touch a lot of code:

          $ find . -name "*.java"|xargs grep "log4j\."|grep "import" -c
          289
          $ find . -name "*.java"|xargs grep "log4j\."|grep "import"|grep "/test/" -c
          242
          
          Show
          wheat9 Haohui Mai added a comment - In our production environment we saw various bottlenecks in NN due to log4j, even with asynchronous logging turned on. Having the option of moving towards log4j2 via slf4j, even it means that it is backward-incompatible is still hugely beneficial. I think it is still possible to pull it off although it will touch a lot of code: $ find . -name "*.java"|xargs grep "log4j\."|grep "import" -c 289 $ find . -name "*.java"|xargs grep "log4j\."|grep "import"|grep "/test/" -c 242
          Hide
          stevel@apache.org Steve Loughran added a comment -

          Happy for you to do the switch to SLF4J APIs; staying in log4j for our tests, properties &c something I'll expect to continue though. That shouldn't impact what you do downstream with the hadoop JARs

          Show
          stevel@apache.org Steve Loughran added a comment - Happy for you to do the switch to SLF4J APIs; staying in log4j for our tests, properties &c something I'll expect to continue though. That shouldn't impact what you do downstream with the hadoop JARs
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1 for moving logging APIs over to slf4j in 2.9.

          Show
          ajisakaa Akira Ajisaka added a comment - +1 for moving logging APIs over to slf4j in 2.9.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Filed HADOOP-14289 for hadoop-common.

          Show
          ajisakaa Akira Ajisaka added a comment - Filed HADOOP-14289 for hadoop-common.

            People

            • Assignee:
              wheat9 Haohui Mai
              Reporter:
              gopalv Gopal V
            • Votes:
              0 Vote for this issue
              Watchers:
              31 Start watching this issue

              Dates

              • Created:
                Updated:

                Development