Log4j 2
  1. Log4j 2
  2. LOG4J2-254

Race condition when setting new filename in RollingFileAppender related code

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0-beta7
    • Fix Version/s: 2.0-beta7
    • Component/s: Appenders
    • Labels:
      None

      Description

      I've come across what very much looks like a race condition in log4j1. In reviewing the log4j 2 code I believe the same condition exists.

      OutputStreamManager.setOutputStream and OutputStreamManager.write need to have a happens-before edge inserted. You could either make OutputStreamManager.os volatile (best) or make setOutputStream synchronized.

      When the RollingFileAppender decides to roll the file it creates a new OutputStream and calls setOutputStream with it. If there is no happens-before edge then that write to OutputStreamManager.os may not be visible to all threads.

      Background:
      I've been attempting to find a way to have applications write logs to Flume without being halted if Flume is down or its channels fill up. My approach was to use the RollingFileAppender from apache-log4j-extras and configure it to roll every minute. Then I setup a Flume spooling directory source to read those files and forward them on.

      I've been having problems with Flume complaining that the rolled log file has changed. The spooling directory source checks this so that people do not attempt to use it on logs that are currently being written to.

      I caught an instance of this this afternoon.

      File: castellan-reader.20130514T2058.log.COMPLETED
      2013-05-14 20:57:05,330 INFO ...

      File: castellan-reader.20130514T2058.log
      2013-05-14 21:23:05,709 DEBUG ...

      Why would an event from 2123 be written into a file from 2058?

      My analysis of log4j 1 code is:
      My understanding of log4j shows that the RollingFileAppenders end up calling this:
      FileAppender:
      public synchronized void setFile(String fileName, boolean append, boolean bufferedIO, int bufferSize)

      Which shortly calls:
      this.qw = new QuietWriter(writer, errorHandler);

      However, the code to actually write to the writer is this:
      protected
      void subAppend(LoggingEvent event) {
      this.qw.write(this.layout.format(event));

      Unless I'm mistaken there's no happens-before edge between setting the qw and calling subappend. The code path to get to subAppend appears not to go through any method synchronized on FileAppender's monitor. this.qw is not volatile.

      Note that I haven't tested log4j 2 for this probable defect - I am raising this work item based on my reading of the code.

        Activity

        Hide
        Edward Sargisson added a comment -

        "OutputStreamManager.os volatile (best)"
        Well, perhaps not best. That would now inject memory fences into client code (and prevent instruction reordering) every time somebody writes to a file appender.

        Show
        Edward Sargisson added a comment - "OutputStreamManager.os volatile (best)" Well, perhaps not best. That would now inject memory fences into client code (and prevent instruction reordering) every time somebody writes to a file appender.
        Hide
        Gary Gregory added a comment -

        Why? The sync point would only happen when the instance variable changes value, not when the object is accessed.

        Show
        Gary Gregory added a comment - Why? The sync point would only happen when the instance variable changes value, not when the object is accessed.
        Hide
        Gary Gregory added a comment -

        Looking at OutputStreamManager, it looks like the footer and header should also be volatile. Alternatively, and preferably, they should be final. In order to make them final, the OutputStreamManager should be passed in the layout, which trickles down into the factory code. The current logic of not overwritting the foot and layout feels like a workaround for the fact that footer and headers are not declared final.

        Show
        Gary Gregory added a comment - Looking at OutputStreamManager, it looks like the footer and header should also be volatile. Alternatively, and preferably, they should be final . In order to make them final, the OutputStreamManager should be passed in the layout, which trickles down into the factory code. The current logic of not overwritting the foot and layout feels like a workaround for the fact that footer and headers are not declared final.
        Hide
        Remko Popma added a comment -

        Edward, nice work on finding this!

        I took a look at OutputStreamManager and I agree that the current code is not correct from a concurrency point of view.
        Specifically, only some methods that access the header, footer or os fields are synchronized, others are not. This means it may break in multi-threaded scenarios.

        One way to fix this would be to synchronize all of the following methods:
        flush()
        close()
        getOutputStream()
        setOutputStream(OutputStream)
        releaseSub()

        Another way is to make the header, footer and os fields volatile, but then the code needs to be modified because code like
        if (this.footer != null)

        { write(this.footer); }

        // as in releaseSub()
        is still broken in concurrent scenarios even if footer is volatile (may give NullPointerExceptions, for example).

        Code like this would need to be replaced by something like this:
        byte[] localVarFooter = this.footer;
        if (localVarFooter != null)

        { write(localVarFooter); }

        I like Gary's suggestion of making header and footer final fields. That would simplify things quite a bit.

        If header and footer are final, and if the os (OutputStream) field is volatile, then synchronization can be removed from the write() method and only the setOutputStream() and close() methods would need a slight modification to avoid problems with concurrent invocations.

        Show
        Remko Popma added a comment - Edward, nice work on finding this! I took a look at OutputStreamManager and I agree that the current code is not correct from a concurrency point of view. Specifically, only some methods that access the header, footer or os fields are synchronized, others are not. This means it may break in multi-threaded scenarios. One way to fix this would be to synchronize all of the following methods: flush() close() getOutputStream() setOutputStream(OutputStream) releaseSub() Another way is to make the header, footer and os fields volatile, but then the code needs to be modified because code like if (this.footer != null) { write(this.footer); } // as in releaseSub() is still broken in concurrent scenarios even if footer is volatile (may give NullPointerExceptions, for example). Code like this would need to be replaced by something like this: byte[] localVarFooter = this.footer; if (localVarFooter != null) { write(localVarFooter); } I like Gary's suggestion of making header and footer final fields. That would simplify things quite a bit. If header and footer are final, and if the os (OutputStream) field is volatile, then synchronization can be removed from the write() method and only the setOutputStream() and close() methods would need a slight modification to avoid problems with concurrent invocations.
        Hide
        Ralph Goers added a comment -

        First, rather than using the RollingFileAppender I suggest you just use the FlumeAppender provided in Log4j 2. Specify type="persistent" or type="embedded", depending on how much of Flume you want to drag into your application.

        As to this problem, yes, the outputstream should be volatile. In the other places where volatile is used I haven't noticed much of a performance hit so I'm not too worried about that.

        I would be very much against having the Manager exposed to Layouts. Actually, that would be impossible since the Layouts are plugins and are created by the configuration before the Appender they will reside in is constructed.

        Obviously the header and footer cannot be declared final since they aren't set in the constructor. I suppose all the OutputStreamAppenders could pass the layout t in their factory methods when they call getManager so that the header and footer could be obtained in the constructor.

        Show
        Ralph Goers added a comment - First, rather than using the RollingFileAppender I suggest you just use the FlumeAppender provided in Log4j 2. Specify type="persistent" or type="embedded", depending on how much of Flume you want to drag into your application. As to this problem, yes, the outputstream should be volatile. In the other places where volatile is used I haven't noticed much of a performance hit so I'm not too worried about that. I would be very much against having the Manager exposed to Layouts. Actually, that would be impossible since the Layouts are plugins and are created by the configuration before the Appender they will reside in is constructed. Obviously the header and footer cannot be declared final since they aren't set in the constructor. I suppose all the OutputStreamAppenders could pass the layout t in their factory methods when they call getManager so that the header and footer could be obtained in the constructor.
        Hide
        Ralph Goers added a comment -

        OutputStream was made volatile and header and footer are final in revision 1482707. Please verify and close.

        Show
        Ralph Goers added a comment - OutputStream was made volatile and header and footer are final in revision 1482707. Please verify and close.
        Hide
        Remko Popma added a comment -

        That was fast!
        Can I suggest 3 small improvements?

        (1)
        setOutputStream (line 95)
        replace:
        this.os.write(header, 0, header.length);
        with:
        os.write(header, 0, header.length);

        (2)
        write(byte[], int, int) (line 110)
        remove (now unnecessary) method synchronization

        (3)
        close() (line 131-135)
        replace:
        if (os == System.out || os == System.err)

        { return; }
        try {
        os.close();
        with:
        OutputStream local = os; // read volatile field only once to avoid potential inconsistent behavior
        if (local == System.out || local == System.err) { return; }

        try {
        local.close();

        If no objections I will commit these tonight after work.

        Show
        Remko Popma added a comment - That was fast! Can I suggest 3 small improvements? (1) setOutputStream (line 95) replace: this.os.write(header, 0, header.length); with: os.write(header, 0, header.length); (2) write(byte[], int, int) (line 110) remove (now unnecessary) method synchronization (3) close() (line 131-135) replace: if (os == System.out || os == System.err) { return; } try { os.close(); with: OutputStream local = os; // read volatile field only once to avoid potential inconsistent behavior if (local == System.out || local == System.err) { return; } try { local.close(); If no objections I will commit these tonight after work.
        Hide
        Gary Gregory added a comment -

        IMO, that is not an improvement. At work we always use "this" to make
        it clear who the receiver is (instance vs static).

        Gary

        Show
        Gary Gregory added a comment - IMO, that is not an improvement. At work we always use "this" to make it clear who the receiver is (instance vs static). Gary
        Hide
        Nick Williams added a comment -

        Yea, not using "this" is a pet peeve of mine, too.

        Show
        Nick Williams added a comment - Yea, not using "this" is a pet peeve of mine, too.
        Hide
        Remko Popma added a comment -

        Gary, there is a subtle difference. It is hard to see from just the code snippet, so here is the method in full:

        protected void setOutputStream(final OutputStream os) {
        this.os = os; // replace value of volatile field
        if (header != null) {
        try

        { this.os.write(header, 0, header.length); // volatile field value may have changed again... }

        catch (final IOException ioe)

        { LOGGER.error("Unable to write header", ioe); }

        }
        }

        Now, imagine a multithreaded scenario where threads t1 and t2 access this method at the same time.
        One possible sequence of events would be:
        [t1] this.os = os; // os is now os1
        [t2] this.os = os; // os is now os2
        [t1] this.os.write(header, 0, header.length); // t1 writes header to os2
        [t2] this.os.write(header, 0, header.length); // t2 writes the same header to os2 again

        In this scenario os1 has no headers written to it and os2 has the same header written to it twice.

        The solution is to call the write method on the local parameter os, not on the field.
        In general, within the same method a volatile field should only be accessed once. Anything else is suspicious.
        (Usage of the volatile field "config" in core.Logger violates this rule of thumb, but that is by design.)

        Show
        Remko Popma added a comment - Gary, there is a subtle difference. It is hard to see from just the code snippet, so here is the method in full: protected void setOutputStream(final OutputStream os) { this.os = os; // replace value of volatile field if (header != null) { try { this.os.write(header, 0, header.length); // volatile field value may have changed again... } catch (final IOException ioe) { LOGGER.error("Unable to write header", ioe); } } } Now, imagine a multithreaded scenario where threads t1 and t2 access this method at the same time. One possible sequence of events would be: [t1] this.os = os; // os is now os1 [t2] this.os = os; // os is now os2 [t1] this.os.write(header, 0, header.length); // t1 writes header to os2 [t2] this.os.write(header, 0, header.length); // t2 writes the same header to os2 again In this scenario os1 has no headers written to it and os2 has the same header written to it twice. The solution is to call the write method on the local parameter os, not on the field. In general, within the same method a volatile field should only be accessed once . Anything else is suspicious. (Usage of the volatile field "config" in core.Logger violates this rule of thumb, but that is by design.)
        Hide
        Nick Williams added a comment -

        Oh. In that case, I think I agree with Remko here. It looks like we should be writing to the local variable.

        For that matter, however, shouldn't the call to this.os = os came AFTER writing to the header? ::

        protected void setOutputStream(final OutputStream os) {
        if (header != null) {
        try

        { os.write(header, 0, header.length); }

        catch (final IOException ioe)

        { LOGGER.error("Unable to write header", ioe); }

        }
        this.os = os;
        }

        Otherwise, events may get written to the output stream before the header.

        Show
        Nick Williams added a comment - Oh. In that case, I think I agree with Remko here. It looks like we should be writing to the local variable. For that matter, however, shouldn't the call to this.os = os came AFTER writing to the header? :: protected void setOutputStream(final OutputStream os) { if (header != null) { try { os.write(header, 0, header.length); } catch (final IOException ioe) { LOGGER.error("Unable to write header", ioe); } } this.os = os; } Otherwise, events may get written to the output stream before the header.
        Hide
        Remko Popma added a comment -

        Nick, I think you are right. Updating the field after writing the header is better.

        Show
        Remko Popma added a comment - Nick, I think you are right. Updating the field after writing the header is better.
        Hide
        Remko Popma added a comment -

        Well then, one more improvement: only update the field if writing to the stream was successful:

        protected void setOutputStream(final OutputStream os) {
        if (header != null) {
        try

        { os.write(header, 0, header.length); this.os = os; // only modify field if the header was successfully written }

        catch (final IOException ioe)

        { LOGGER.error("Unable to write header", ioe); }

        }
        }

        Show
        Remko Popma added a comment - Well then, one more improvement: only update the field if writing to the stream was successful: protected void setOutputStream(final OutputStream os) { if (header != null) { try { os.write(header, 0, header.length); this.os = os; // only modify field if the header was successfully written } catch (final IOException ioe) { LOGGER.error("Unable to write header", ioe); } } }
        Hide
        Ralph Goers added a comment -

        Remko, in practical terms what you are describing won't happen. RollingFileManager needs to insure that the file is being rolled by only a single thread or everything will be a mess. Notice that the setOutputStream is only ever called from methods that are within synchronized constructs. So the change in (1) is not needed.

        Removing the synchronization on the write method as you suggest in (2) may break things. Look at the comments on the method. We aren't synchronizing because the value of os may change but because the OutputStream itself might not be thread safe.

        The third recommendation, while technically correct, has the problem that if you have multiple threads calling close something is very broken. Again, only one thread should ever attempt to close the stream. However, I don't see any harm in making this change.

        Show
        Ralph Goers added a comment - Remko, in practical terms what you are describing won't happen. RollingFileManager needs to insure that the file is being rolled by only a single thread or everything will be a mess. Notice that the setOutputStream is only ever called from methods that are within synchronized constructs. So the change in (1) is not needed. Removing the synchronization on the write method as you suggest in (2) may break things. Look at the comments on the method. We aren't synchronizing because the value of os may change but because the OutputStream itself might not be thread safe. The third recommendation, while technically correct, has the problem that if you have multiple threads calling close something is very broken. Again, only one thread should ever attempt to close the stream. However, I don't see any harm in making this change.
        Hide
        Ralph Goers added a comment -

        The change you propose to setOutputStream above does seem reasonable.

        Show
        Ralph Goers added a comment - The change you propose to setOutputStream above does seem reasonable.
        Hide
        Remko Popma added a comment -

        Ralph, you are right about the synchronization. I missed that access to the stream itself needs to be synchronized.

        About the volatile field usage (1 and 3), I was just looking at this from a multi-threading perspective and these are just noticeable little anti-patterns that caught my eye.
        I realize that with current usage nobody will notice any issue.

        However I would still argue that it is better to fix them, if only because OutputStreamManager acts like a base class and future subclasses (or custom subclasses by users) may have different concurrency usage. (I'm just in the habit of "only one access to a volatile field per method". Just because I'm paranoid, doesn't mean they aren't out to get me.... )

        Show
        Remko Popma added a comment - Ralph, you are right about the synchronization. I missed that access to the stream itself needs to be synchronized. About the volatile field usage (1 and 3), I was just looking at this from a multi-threading perspective and these are just noticeable little anti-patterns that caught my eye. I realize that with current usage nobody will notice any issue. However I would still argue that it is better to fix them, if only because OutputStreamManager acts like a base class and future subclasses (or custom subclasses by users) may have different concurrency usage. (I'm just in the habit of "only one access to a volatile field per method". Just because I'm paranoid, doesn't mean they aren't out to get me.... )
        Hide
        Remko Popma added a comment -

        Improvements (1) and (3) are now committed to trunk.

        Show
        Remko Popma added a comment - Improvements (1) and (3) are now committed to trunk.
        Hide
        Edward Sargisson added a comment -

        Hi all,
        Thank you for looking at this defect and fixing it so quickly. I've had a look and I'm not familiar enough with log4j2 code but it looks like setOutputStream(( will refuse to set the os field if header is null. AbstractLayout appears to be where header is stored and it states that header may be null. The header == null probably just needs to be this.os = os;

        Show
        Edward Sargisson added a comment - Hi all, Thank you for looking at this defect and fixing it so quickly. I've had a look and I'm not familiar enough with log4j2 code but it looks like setOutputStream(( will refuse to set the os field if header is null. AbstractLayout appears to be where header is stored and it states that header may be null. The header == null probably just needs to be this.os = os;
        Hide
        Edward Sargisson added a comment - - edited

        This code doesn't work the second time the file rolls. I get the following stack trace:

        2013-05-22 22:24:09,522 ERROR Unable to write to stream /var/log/flume-ng/flume.log for appender RollingFile
        2013-05-22 22:24:09,522 ERROR An exception occurred processing Appender RollingFile org.apache.logging.log4j.core.appender.AppenderRuntimeException: Error flushing stream /var/log/flume-ng/flume.log
        at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:150)
        at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:115)
        at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:84)
        at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:102)
        at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:404)
        at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:385)
        at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:346)
        at org.apache.logging.log4j.core.Logger.log(Logger.java:110)
        at org.apache.logging.log4j.spi.AbstractLoggerWrapper.log(AbstractLoggerWrapper.java:55)
        at org.slf4j.impl.SLF4JLogger.warn(SLF4JLogger.java:311)
        at org.elasticsearch.common.logging.slf4j.Slf4jESLogger.internalWarn(Slf4jESLogger.java:104)
        at org.elasticsearch.common.logging.support.AbstractESLogger.warn(AbstractESLogger.java:100)
        at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:319)
        at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:281)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
        Caused by: java.io.IOException: Stream Closed
        at java.io.FileOutputStream.writeBytes(Native Method)
        at java.io.FileOutputStream.write(FileOutputStream.java:318)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
        at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:147)
        ... 16 more

        This is from a log4j2.xml with the following snippet:
        <RollingFile name="RollingFile" fileName="/var/log/flume-ng/flume.log"
        filePattern="/var/log/flume-ng/spool/flume.%d

        {yyyyMMdd'T'HHmm}

        .log">

        <!-- date level [thread] logger message exception newline -->
        <PatternLayout pattern="%d %5p [%t] %25c

        {1.}

        %m%xEx%n" />
        <Policies>
        <TimeBasedTriggeringPolicy />
        </Policies>
        <DefaultRolloverStrategy max="20"/>
        </RollingFile>

        Since this now breaks logging, and because I can't find a way to set the header so there appears to be no workaround, I've taken the liberty of setting this defect to critical.

        Show
        Edward Sargisson added a comment - - edited This code doesn't work the second time the file rolls. I get the following stack trace: 2013-05-22 22:24:09,522 ERROR Unable to write to stream /var/log/flume-ng/flume.log for appender RollingFile 2013-05-22 22:24:09,522 ERROR An exception occurred processing Appender RollingFile org.apache.logging.log4j.core.appender.AppenderRuntimeException: Error flushing stream /var/log/flume-ng/flume.log at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:150) at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:115) at org.apache.logging.log4j.core.appender.RollingFileAppender.append(RollingFileAppender.java:84) at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:102) at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:404) at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:385) at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:346) at org.apache.logging.log4j.core.Logger.log(Logger.java:110) at org.apache.logging.log4j.spi.AbstractLoggerWrapper.log(AbstractLoggerWrapper.java:55) at org.slf4j.impl.SLF4JLogger.warn(SLF4JLogger.java:311) at org.elasticsearch.common.logging.slf4j.Slf4jESLogger.internalWarn(Slf4jESLogger.java:104) at org.elasticsearch.common.logging.support.AbstractESLogger.warn(AbstractESLogger.java:100) at org.elasticsearch.client.transport.TransportClientNodesService$SimpleNodeSampler.sample(TransportClientNodesService.java:319) at org.elasticsearch.client.transport.TransportClientNodesService$ScheduledNodeSampler.run(TransportClientNodesService.java:281) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) Caused by: java.io.IOException: Stream Closed at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:318) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:147) ... 16 more This is from a log4j2.xml with the following snippet: <RollingFile name="RollingFile" fileName="/var/log/flume-ng/flume.log" filePattern="/var/log/flume-ng/spool/flume.%d {yyyyMMdd'T'HHmm} .log"> <!-- date level [thread] logger message exception newline --> <PatternLayout pattern="%d %5p [%t] %25c {1.} %m%xEx%n" /> <Policies> <TimeBasedTriggeringPolicy /> </Policies> <DefaultRolloverStrategy max="20"/> </RollingFile> Since this now breaks logging, and because I can't find a way to set the header so there appears to be no workaround, I've taken the liberty of setting this defect to critical.
        Hide
        Edward Sargisson added a comment -

        I have manually patched my copy of log4j2 and confirmed the defect and fix mentioned above. Without this fix log4j2 will rollover precisely once and then stop logging.

        OutputStreeamManager.setOutputStream needs to be as follows:
        protected void setOutputStream(final OutputStream os) {
        if (header != null) {
        try

        { os.write(header, 0, header.length); this.os = os; // only update field if os.write() succeeded }

        catch (final IOException ioe)

        { LOGGER.error("Unable to write header", ioe); }

        } else

        { this.os = os; }

        }

        Show
        Edward Sargisson added a comment - I have manually patched my copy of log4j2 and confirmed the defect and fix mentioned above. Without this fix log4j2 will rollover precisely once and then stop logging. OutputStreeamManager.setOutputStream needs to be as follows: protected void setOutputStream(final OutputStream os) { if (header != null) { try { os.write(header, 0, header.length); this.os = os; // only update field if os.write() succeeded } catch (final IOException ioe) { LOGGER.error("Unable to write header", ioe); } } else { this.os = os; } }
        Hide
        Ralph Goers added a comment -

        Thanks for catching that! Fix was applied in revision 1486421.

        Show
        Ralph Goers added a comment - Thanks for catching that! Fix was applied in revision 1486421.
        Hide
        Ralph Goers added a comment -

        Please verify and close.

        Show
        Ralph Goers added a comment - Please verify and close.
        Hide
        Edward Sargisson added a comment -

        I have verified the code by inspection - and by running a test that fails unless the new code is used.

        Verified and closing.

        Show
        Edward Sargisson added a comment - I have verified the code by inspection - and by running a test that fails unless the new code is used. Verified and closing.

          People

          • Assignee:
            Unassigned
            Reporter:
            Edward Sargisson
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development