[MINIFI-390] Logback archiving can cause issues with rollover when compression is enabled

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.2.0
• Component/s: Core Framework
• Assignee: Unassigned
• Reporter: Aldrin Piri
• Created: 21/Aug/17 16:55
• Updated: 15/Dec/17 20:25
• Resolved: 15/Dec/17 20:25

Description

At high rates of logging, the compression of archived log files configured in logback can cause an unbounded growth of file size despite totalSizeCap.

The issue can be seen in the following APP appender and triggered with a root-level of DEBUG.

    <appender name="APP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${org.apache.nifi.minifi.bootstrap.config.log.dir}/minifi-app.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <!--
              For daily rollover, use 'app_%d.log'.
              For hourly rollover, use 'app_%d{yyyy-MM-dd_HH}.log'.
              To GZIP rolled files, replace '.log' with '.log.gz'.
              To ZIP rolled files, replace '.log' with '.log.zip'.
            -->
            <fileNamePattern>${org.apache.nifi.minifi.bootstrap.config.log.dir}/minifi-app_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
            <!-- Keep 10 rolling periods worth of log files-->
            <maxHistory>1</maxHistory>
            <!-- Max size each log file will be-->
            <maxFileSize>10MB</maxFileSize>
            <!-- Provide a cap of 10 MB across all archive files -->
            <totalSizeCap>1000MB</totalSizeCap>
        </rollingPolicy>
        <immediateFlush>true</immediateFlush>
        <encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
            <pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
        </encoder>
    </appender>

This seems to be related to https://jira.qos.ch/browse/LOGBACK-1162 which is currently in progress. We should monitor this progression and incorporate the updated dependency when released. The workaround for those environments experiencing this is to disable archiving (by removing the .gz from the filename pattern).

Issue Links

• Is contained by: NIFI-8593 Upgrade to newer version of logback