Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4431

Add a way to control the number of audit event log files

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
    • Environment:
      Red Hat Enterprise Linux Server release 6.5

      Description

      There is no limit to the number of audit event log files. When audit event log is enabled, the growing number of log files will spill the disk space.
      So, the number of log files must be controlled by way of configuration as general log file number controlling.

        Activity

        Hide
        davidxdh Donghui Xu added a comment -

        OK. I didn't find my patch in the link you provided. I have changed the Fix Version to 2.9.0.

        Show
        davidxdh Donghui Xu added a comment - OK. I didn't find my patch in the link you provided. I have changed the Fix Version to 2.9.0.
        Hide
        jbapple Jim Apple added a comment -

        This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09.

        2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that:

        https://github.com/apache/incubator-impala/commits/2.8.0

        Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0?

        Thank you!

        Show
        jbapple Jim Apple added a comment - This is a bulk comment on all issues with Fix Version 2.8.0 that were resolved on or after 2016-12-09. 2.8.0 was branched on December 9, with only two changes to master cherry-picked to the 2.8.0 release branch after that: https://github.com/apache/incubator-impala/commits/2.8.0 Issues fixed after December 9 might not be fixed in 2.8.0. If you are the one who marked this issue Resolved, can you check to see if the patch is in 2.8.0 by using the link above? If the patch is not in 2.8.0, can you change the Fix Version to 2.9.0? Thank you!
        Hide
        davidxdh Donghui Xu added a comment -

        IMPALA-4431: Add audit event log control mechanism to prevent disk
        overflow

        There is no limit to the number of audit event log files. When audit
        event log is enabled the growing number of log files will fill up the
        disk space.

        This patch adds checking and rotation mechanism on audit event log
        files to prevent file number out of control which causes disk overflow.

        Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368
        Reviewed-on: http://gerrit.cloudera.org:8080/4971
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        davidxdh Donghui Xu added a comment - IMPALA-4431 : Add audit event log control mechanism to prevent disk overflow There is no limit to the number of audit event log files. When audit event log is enabled the growing number of log files will fill up the disk space. This patch adds checking and rotation mechanism on audit event log files to prevent file number out of control which causes disk overflow. Change-Id: I8c3229cbdb6275f969c15258c9ccab6efeb24368 Reviewed-on: http://gerrit.cloudera.org:8080/4971 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins
        Hide
        lv Lars Volker added a comment -

        Greg Rahn - Yes, that's the idea. This can be used to prevent the audit files from filling the disk. Thanks for the clarification.

        Show
        lv Lars Volker added a comment - Greg Rahn - Yes, that's the idea. This can be used to prevent the audit files from filling the disk. Thanks for the clarification.
        Hide
        grahn Greg Rahn added a comment -

        Lars Volker - I'd say the default should be infinite to match current behavior such that upgrades do not change behavior w/o an explicit action. Just to clarify, the ask is a parameter/behavior for audit logs to match that of max_log_files for regular logs, correct?

        Show
        grahn Greg Rahn added a comment - Lars Volker - I'd say the default should be infinite to match current behavior such that upgrades do not change behavior w/o an explicit action. Just to clarify, the ask is a parameter/behavior for audit logs to match that of max_log_files for regular logs, correct?
        Hide
        davidxdh Donghui Xu added a comment -

        I think you are right. I have changed default number of audit log files to retain to 0 which means that all log files are retained by default.

        Show
        davidxdh Donghui Xu added a comment - I think you are right. I have changed default number of audit log files to retain to 0 which means that all log files are retained by default.
        Hide
        lv Lars Volker added a comment -

        Greg Rahn - If we want to add audit log file rotation, does it make sense to have a finite, default number of files to retain? Or do we have to retain all files by default to prevent conflicts with whatever external retention policy a user might have for audit log?

        Show
        lv Lars Volker added a comment - Greg Rahn - If we want to add audit log file rotation, does it make sense to have a finite, default number of files to retain? Or do we have to retain all files by default to prevent conflicts with whatever external retention policy a user might have for audit log?

          People

          • Assignee:
            davidxdh Donghui Xu
            Reporter:
            davidxdh Donghui Xu
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development