Hive
  1. Hive
  2. HIVE-1948

Have audit logging in the Metastore

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.7.0
    • Component/s: Logging, Metastore, Security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      It would be good to have audit logging in the metastore, similar to Hadoop's NameNode audit logging. This would allow administrators to dig into details about which user performed metadata operations (like create/drop tables/partitions) and from where (IP address).

      1. audit-log-3.patch
        6 kB
        Devaraj Das
      2. audit-log-2.patch
        7 kB
        Devaraj Das
      3. audit-log.1.patch
        5 kB
        Devaraj Das
      4. audit-log.patch
        5 kB
        Devaraj Das

        Issue Links

          Activity

          Devaraj Das created issue -
          Hide
          Devaraj Das added a comment -

          Early patch. I am testing it.

          Show
          Devaraj Das added a comment - Early patch. I am testing it.
          Devaraj Das made changes -
          Field Original Value New Value
          Attachment audit-log.patch [ 12470089 ]
          Hide
          Devaraj Das added a comment -

          A slightly updated patch.

          Show
          Devaraj Das added a comment - A slightly updated patch.
          Devaraj Das made changes -
          Attachment audit-log.1.patch [ 12470217 ]
          Hide
          Devaraj Das added a comment -

          Submitting patch for review. There is one caveat with this patch - it won't log the IP address of the remote clients when security is enabled in Hive. Making this work means a change in thrift. I have raised THRIFT-1053 for the same. Once THRIFT-1053 is addressed, I will provide a fix (in a different jira) to capture the IP address for the secure case too.

          Show
          Devaraj Das added a comment - Submitting patch for review. There is one caveat with this patch - it won't log the IP address of the remote clients when security is enabled in Hive. Making this work means a change in thrift. I have raised THRIFT-1053 for the same. Once THRIFT-1053 is addressed, I will provide a fix (in a different jira) to capture the IP address for the secure case too.
          Devaraj Das made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Devaraj Das added a comment -

          https://reviews.apache.org/r/398/ is the reviewboard URL

          Show
          Devaraj Das added a comment - https://reviews.apache.org/r/398/ is the reviewboard URL
          Hide
          Namit Jain added a comment -

          What about the performance impact for this ?
          I mean, there seems to be no way to turn it off - is reading the conf.

          try

          { ugi = ShimLoader.getHadoopShims().getUGIForConf(getConf()); }

          catch (Exception ex)

          { throw new RuntimeException(ex); }

          for every audit operation acceptable ?

          Show
          Namit Jain added a comment - What about the performance impact for this ? I mean, there seems to be no way to turn it off - is reading the conf. try { ugi = ShimLoader.getHadoopShims().getUGIForConf(getConf()); } catch (Exception ex) { throw new RuntimeException(ex); } for every audit operation acceptable ?
          Hide
          Devaraj Das added a comment -

          In this patch I made the logging to happen only when a secure shim is deployed (the case where such audit logging makes most sense).

          Show
          Devaraj Das added a comment - In this patch I made the logging to happen only when a secure shim is deployed (the case where such audit logging makes most sense).
          Devaraj Das made changes -
          Attachment audit-log-2.patch [ 12470518 ]
          Hide
          Namit Jain added a comment -

          Can you regenerate the patch - I am getting some merge conflicts.

          Show
          Namit Jain added a comment - Can you regenerate the patch - I am getting some merge conflicts.
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Devaraj Das added a comment -

          Regenerated patch

          Show
          Devaraj Das added a comment - Regenerated patch
          Devaraj Das made changes -
          Attachment audit-log-3.patch [ 12470680 ]
          Devaraj Das made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Namit Jain added a comment -

          +1

          Show
          Namit Jain added a comment - +1
          Hide
          Namit Jain added a comment -

          Committed. Thanks Devaraj

          Show
          Namit Jain added a comment - Committed. Thanks Devaraj
          Namit Jain made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Carl Steinbach made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Carl Steinbach made changes -
          Component/s Logging [ 12312594 ]
          Component/s Security [ 12313866 ]
          Carl Steinbach made changes -
          Link This issue relates HIVE-3277 [ HIVE-3277 ]
          Gavin made changes -
          Link This issue relates to HIVE-3277 [ HIVE-3277 ]
          Gavin made changes -
          Link This issue relates to HIVE-3277 [ HIVE-3277 ]
          Lefty Leverenz made changes -
          Link This issue relates to HIVE-3505 [ HIVE-3505 ]
          Lefty Leverenz made changes -
          Link This issue relates to HIVE-5988 [ HIVE-5988 ]
          Lefty Leverenz made changes -
          Link This issue relates to HIVE-2797 [ HIVE-2797 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Patch Available Patch Available Open Open
          4d 7h 41m 1 Namit Jain 09/Feb/11 06:56
          Open Open Patch Available Patch Available
          2d 9h 10m 2 Devaraj Das 09/Feb/11 16:10
          Patch Available Patch Available Resolved Resolved
          4h 52m 1 Namit Jain 09/Feb/11 21:02
          Resolved Resolved Closed Closed
          310d 2h 58m 1 Carl Steinbach 17/Dec/11 00:01

            People

            • Assignee:
              Devaraj Das
              Reporter:
              Devaraj Das
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development