Hadoop Common
  1. Hadoop Common
  2. HADOOP-6586

Log authentication and authorization failures and successes

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This jira will cover RPC authentication and SL authorizations logging.

      1. HADOOP-6586-8-BP20-1.patch
        11 kB
        Boris Shkolnik
      2. HADOOP-6586-8-BP20.patch
        11 kB
        Boris Shkolnik
      3. HADOOP-6586-8.patch
        9 kB
        Boris Shkolnik
      4. HADOOP-6586-7.patch
        9 kB
        Boris Shkolnik
      5. HADOOP-6586-5.patch
        20 kB
        Boris Shkolnik
      6. HADOOP-6586-4.patch
        6 kB
        Boris Shkolnik
      7. HADOOP-6586-3.patch
        6 kB
        Boris Shkolnik
      8. HADOOP-6586.patch
        4 kB
        Boris Shkolnik

        Activity

        Hide
        Allen Wittenauer added a comment -

        Awesome. Now I can make my "Boris Can't Spell" JIRA.

        Show
        Allen Wittenauer added a comment - Awesome. Now I can make my "Boris Can't Spell" JIRA.
        Hide
        Chris Douglas added a comment -

        Setting resolution to fixed, as the patch was committed and not reverted.

        Show
        Chris Douglas added a comment - Setting resolution to fixed, as the patch was committed and not reverted.
        Hide
        Amar Kamat added a comment -

        Boris,

        • If we want to model it as audit log, then should keep the naming consistent with hdfs i.e classname.audit?
        • I am working on MAPREDUCE-1543 and I have proposed a mapreduce friendly audit log format. Surely we dont want 2 audit-log formats. Here is my proposal. Lets discuss it out. If the commons audit-log format turns out to be different from mapreduce, then there should be some way to distinguish them.
        Show
        Amar Kamat added a comment - Boris, If we want to model it as audit log, then should keep the naming consistent with hdfs i.e classname.audit? I am working on MAPREDUCE-1543 and I have proposed a mapreduce friendly audit log format. Surely we dont want 2 audit-log formats. Here is my proposal . Lets discuss it out. If the commons audit-log format turns out to be different from mapreduce, then there should be some way to distinguish them.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #193 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/193/)
        . Log authentication and authorization failures and successes for RPC

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #193 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/193/ ) . Log authentication and authorization failures and successes for RPC
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #262 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/262/)
        . Log authentication and authorization failures and successes for RPC

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #262 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/262/ ) . Log authentication and authorization failures and successes for RPC
        Hide
        Boris Shkolnik added a comment -

        merged with branch changes.

        Show
        Boris Shkolnik added a comment - merged with branch changes.
        Hide
        Boris Shkolnik added a comment -

        HADOOP-6586-8-BP20.patch - for previous version. Not for commit

        Show
        Boris Shkolnik added a comment - HADOOP-6586 -8-BP20.patch - for previous version. Not for commit
        Hide
        Allen Wittenauer added a comment -

        OK, then I'm going to re-open this.

        The logging format should be consistent between the two, with proper identifiers in place to say whether this is a user or an internal protocol. Free form text == death here. I'd like to propose the following:

        date INFO service: Auth [successful|failed] for [identify] using [protocol=protocolName|user=username]

        IIRC, we support user remapping, so identifying which identity is being used for which user would be helpful here.

        Show
        Allen Wittenauer added a comment - OK, then I'm going to re-open this. The logging format should be consistent between the two, with proper identifiers in place to say whether this is a user or an internal protocol. Free form text == death here. I'd like to propose the following: date INFO service: Auth [successful|failed] for [identify] using [protocol=protocolName|user=username] IIRC, we support user remapping, so identifying which identity is being used for which user would be helpful here.
        Hide
        Boris Shkolnik added a comment -

        first line is authentication of the user
        second line is authorization for a specific protocol

        For failure "successful => failed"

        Show
        Boris Shkolnik added a comment - first line is authentication of the user second line is authorization for a specific protocol For failure "successful => failed"
        Hide
        Allen Wittenauer added a comment -

        That's pretty awful to parse if I want to do a security audit.

        Do the two lines actually signify the same connection was successful in two different parts of the stack?

        What does failed look like?

        Show
        Allen Wittenauer added a comment - That's pretty awful to parse if I want to do a security audit. Do the two lines actually signify the same connection was successful in two different parts of the stack? What does failed look like?
        Hide
        Boris Shkolnik added a comment -

        No, but now I know there is one.
        Here is sample of the output:
        2010-02-26 09:48:04,997 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successfull for ssl/fake@HADOOP.ORG
        2010-02-26 09:48:04,998 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successfull for ssl/fake@HADOOP.ORG for protocol=interface org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol

        If you have a different suggestion - please propose it. I can look into this later.

        Show
        Boris Shkolnik added a comment - No, but now I know there is one. Here is sample of the output: 2010-02-26 09:48:04,997 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successfull for ssl/fake@HADOOP.ORG 2010-02-26 09:48:04,998 INFO SecurityLogger.org.apache.hadoop.security.authorize.ServiceAuthorizationManager: Authorization successfull for ssl/fake@HADOOP.ORG for protocol=interface org.apache.hadoop.hdfs.server.protocol.NamenodeProtocol If you have a different suggestion - please propose it. I can look into this later.
        Hide
        Allen Wittenauer added a comment -

        Is the reason you didn't post output because you have a spelling mistake?

        Show
        Allen Wittenauer added a comment - Is the reason you didn't post output because you have a spelling mistake?
        Hide
        Boris Shkolnik added a comment -

        commited this.

        Show
        Boris Shkolnik added a comment - commited this.
        Hide
        Boris Shkolnik added a comment -

        addressed Kan's comment.

        Show
        Boris Shkolnik added a comment - addressed Kan's comment.
        Hide
        Allen Wittenauer added a comment -

        Can we see some sample output please? Again, I stress: this log needs to be separate-able and parse-able similar to the audit log.

        Show
        Allen Wittenauer added a comment - Can we see some sample output please? Again, I stress: this log needs to be separate-able and parse-able similar to the audit log.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12437091/HADOOP-6586-7.patch
        against trunk revision 916529.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/390/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/390/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/390/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/390/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437091/HADOOP-6586-7.patch against trunk revision 916529. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/390/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/390/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/390/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/390/console This message is automatically generated.
        Hide
        Kan Zhang added a comment -

        otherwise, +1 on the patch.

        Show
        Kan Zhang added a comment - otherwise, +1 on the patch.
        Hide
        Kan Zhang added a comment -

        you don't want to catch the exception thrown by tokenIdentifier.getUser() and swallow it.

        Show
        Kan Zhang added a comment - you don't want to catch the exception thrown by tokenIdentifier.getUser() and swallow it.
        Hide
        Boris Shkolnik added a comment -

        merged with trunk

        Show
        Boris Shkolnik added a comment - merged with trunk
        Hide
        Boris Shkolnik added a comment -

        added Connection member to the Handle, to be able to record attempting user.

        Show
        Boris Shkolnik added a comment - added Connection member to the Handle, to be able to record attempting user.
        Hide
        Boris Shkolnik added a comment -

        good point. On Kan's suggestion changed it to use clients ip:port for logging
        (because we don't have any authenticated name in the case of failure).

        Show
        Boris Shkolnik added a comment - good point. On Kan's suggestion changed it to use clients ip:port for logging (because we don't have any authenticated name in the case of failure).
        Hide
        Kan Zhang added a comment -

        Or IllegalStateException.

        Show
        Kan Zhang added a comment - Or IllegalStateException.
        Hide
        Kan Zhang added a comment -

        When authentication fails, saslServer.getAuthorizationID() will give you null.

        Show
        Kan Zhang added a comment - When authentication fails, saslServer.getAuthorizationID() will give you null.
        Hide
        Boris Shkolnik added a comment -

        auth logs only in secure mode
        Creates a separate log file SecurityAuth.audit (Daily Roll)
        I've tested some cases manually.
        Not sure about auth fail tests.

        Show
        Boris Shkolnik added a comment - auth logs only in secure mode Creates a separate log file SecurityAuth.audit (Daily Roll) I've tested some cases manually. Not sure about auth fail tests.
        Hide
        Allen Wittenauer added a comment -

        Security logging needs to be done similarly to HDFS audit logging. It needs to be easily separate-able and parse-able. It should have a fairly static format so that tools won't break from release to release.

        Show
        Allen Wittenauer added a comment - Security logging needs to be done similarly to HDFS audit logging. It needs to be easily separate-able and parse-able. It should have a fairly static format so that tools won't break from release to release.
        Hide
        Boris Shkolnik added a comment -

        preliminary patch for discussion.

        Show
        Boris Shkolnik added a comment - preliminary patch for discussion.

          People

          • Assignee:
            Boris Shkolnik
            Reporter:
            Boris Shkolnik
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development