Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1315

Add fsck event to audit log and remove other audit log events corresponding to FSCK listStatus and open calls

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: namenode, tools
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      When running fsck, audit log events are not logged for listStatus and open are not logged. A new event with cmd=fsck is logged with ugi field set to the user requesting fsck and src field set to the fsck path.

      Description

      In Yahoo production cluster audit logs more than 50% of entries comes from FSCK. FSCK makes same Namenode method calls as clients. Because of this, the operations are logged in audit logs similar to operations invoked by the client. I want to make a change to namenode not to log operations invoked by FSCK in audit logs.

      1. HDFS-1315.patch
        10 kB
        Suresh Srinivas
      2. HDFS-1315.1.patch
        18 kB
        Suresh Srinivas
      3. HDFS-1315.y20.patch
        18 kB
        Suresh Srinivas

        Issue Links

          Activity

          Hide
          Suresh Srinivas added a comment -

          This has many benefits, especially in setups where FSCK is run daily:

          1. Access log no longer has the noise related to FSCK invoked operations.
          2. Access log shrink by more than 50%.
          3. Namenode no longer generates garbage related to access logs (close 13GB for a system with 60million files), resulting in less work done by Namenode java process to collect garbage.
          Show
          Suresh Srinivas added a comment - This has many benefits, especially in setups where FSCK is run daily: Access log no longer has the noise related to FSCK invoked operations. Access log shrink by more than 50%. Namenode no longer generates garbage related to access logs (close 13GB for a system with 60million files), resulting in less work done by Namenode java process to collect garbage.
          Hide
          Allen Wittenauer added a comment -

          I'd say HDFS-1316 blocks this issue.

          The audit log MUST have an entry for fsck.

          Show
          Allen Wittenauer added a comment - I'd say HDFS-1316 blocks this issue. The audit log MUST have an entry for fsck.
          Hide
          Suresh Srinivas added a comment -

          FSCK when get block locations, the block access tokens are unnecessarily created. Based on a test run I found without block access tokens, the FSNamesystem#getBlockLocations() method is more than 30% faster.

          For 10,000 invocations of getBlockLocations():

          Number of blocks in file With access tokens Without access tokens
          1 2154 1702
          10 2183 1753
          100 2771 2007
          1000 9047 5757

          I also intend to call from FSCK the variant of getBlockLocations() without block access tokens.

          Show
          Suresh Srinivas added a comment - FSCK when get block locations, the block access tokens are unnecessarily created. Based on a test run I found without block access tokens, the FSNamesystem#getBlockLocations() method is more than 30% faster. For 10,000 invocations of getBlockLocations(): Number of blocks in file With access tokens Without access tokens 1 2154 1702 10 2183 1753 100 2771 2007 1000 9047 5757 I also intend to call from FSCK the variant of getBlockLocations() without block access tokens.
          Hide
          Suresh Srinivas added a comment -

          Attached patch makes the following changes:

          1. Audit log is not logged for invocation of methods internal to the namenode.
          2. FSCK gets block locations without modifying access time of the file and without block access tokens.
          Show
          Suresh Srinivas added a comment - Attached patch makes the following changes: Audit log is not logged for invocation of methods internal to the namenode. FSCK gets block locations without modifying access time of the file and without block access tokens.
          Hide
          Konstantin Shvachko added a comment -

          isInternalInvocation() is used only with negation. So may be we should change it to isExternalInvocation() and save on negations.
          Other than that the patch looks good to me.

          Show
          Konstantin Shvachko added a comment - isInternalInvocation() is used only with negation. So may be we should change it to isExternalInvocation() and save on negations. Other than that the patch looks good to me.
          Hide
          Suresh Srinivas added a comment -

          Additional changes in the patch:

          1. Added an audit log event for fsck
          2. Added tests for this change to make sure only audit log only has fsck event.

          I will close HDFS-1316, as we are adding fsck audit log event as a part of this change.

          Show
          Suresh Srinivas added a comment - Additional changes in the patch: Added an audit log event for fsck Added tests for this change to make sure only audit log only has fsck event. I will close HDFS-1316 , as we are adding fsck audit log event as a part of this change.
          Hide
          Suresh Srinivas added a comment -

          Updated patch.

          Show
          Suresh Srinivas added a comment - Updated patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12450287/HDFS-1315.1.patch
          against trunk revision 966952.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 7 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/445/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/445/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/445/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/445/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12450287/HDFS-1315.1.patch against trunk revision 966952. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/445/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/445/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/445/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/445/console This message is automatically generated.
          Hide
          Suresh Srinivas added a comment -

          Failed tests are unrelated to this patch:
          TestFileAppend4 tracked in HDFS-1306
          TestBlockReport tracked in HDFS-733
          TestFiHFlush tracked in HDFS-1206
          TestFileConcurrentReader tracked in HDFS-1310

          Show
          Suresh Srinivas added a comment - Failed tests are unrelated to this patch: TestFileAppend4 tracked in HDFS-1306 TestBlockReport tracked in HDFS-733 TestFiHFlush tracked in HDFS-1206 TestFileConcurrentReader tracked in HDFS-1310
          Hide
          Konstantin Shvachko added a comment -

          +1. Patch looks good.
          Could you please file test failures that don't have jiras yet.

          Show
          Konstantin Shvachko added a comment - +1. Patch looks good. Could you please file test failures that don't have jiras yet.
          Hide
          Allen Wittenauer added a comment -

          Is fsck audit logging just "hey fsck was run!" or does it say which files/dirs was specified against the fsck?

          Show
          Allen Wittenauer added a comment - Is fsck audit logging just "hey fsck was run!" or does it say which files/dirs was specified against the fsck?
          Hide
          Suresh Srinivas added a comment -

          > Could you please file test failures that don't have jiras yet.
          All test failures are tracked in my previous comment

          Show
          Suresh Srinivas added a comment - > Could you please file test failures that don't have jiras yet. All test failures are tracked in my previous comment
          Hide
          Suresh Srinivas added a comment -

          y20 version of the patch

          Show
          Suresh Srinivas added a comment - y20 version of the patch
          Hide
          Suresh Srinivas added a comment -

          Its an audit log entry - has all the audit log entry information such as ugi with user information, ipaddress of the requestor, fsck path in src field.

          Show
          Suresh Srinivas added a comment - Its an audit log entry - has all the audit log entry information such as ugi with user information, ipaddress of the requestor, fsck path in src field.
          Hide
          Suresh Srinivas added a comment -

          I committed this change to trunk.

          Show
          Suresh Srinivas added a comment - I committed this change to trunk.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #352 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/352/)
          HDFS-1315. Add fsck event to audit log and remove other audit log events corresponding to FSCK listStatus and open calls. Contributed by Suresh Srinivas.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #352 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/352/ ) HDFS-1315 . Add fsck event to audit log and remove other audit log events corresponding to FSCK listStatus and open calls. Contributed by Suresh Srinivas.

            People

            • Assignee:
              Suresh Srinivas
              Reporter:
              Suresh Srinivas
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development