I tested this log format change using a persistent store, with one producer and one consumer and a transaction batch size of one, again with logger org.apache.qpid set to DEBUG. I found that it still causes a significant performance improvement - throughput doubled on my laptop.
My laptop is a cheap dual core machine with an SSD drive, so it probably exaggerates the impact of this change, due to its fast IO and slow processor. Nevertheless, these results suggest to me that typical production deployments would see a significant performance increase when using DEBUG logging.
When the log level is INFO or above, there is, as expected, no measurable performance improvement.
The scenario I have in mind is when a production support team want to switch on DEBUG logging for a few minutes to help pin down the source of a performance problem, and obviously want to minimise the impact of this logging. It is frustrating when support teams are so fearful of the performance impact of debug-level logging that they resist enabling it, even if it would help solve a problem.
Regarding line numbers... on the whole, our log messages are sufficiently descriptive to allow us to locate the line of code that produced them, without requiring file names and line numbers. In my short time working on Qpid, I've never had to resort to using line numbers (in fact, they can be positively misleading in cases where we've written our own centralised logging functions). Any logging that is untraceable without an explicit file name and line number is obviously inadequate and should be fixed.
My view is that the performance increase of this change offsets the lack of line numbers in the log file. However, I accept that this is a trade-off so won't push this change if the majority disagree with me.