Flume
  1. Flume
  2. FLUME-2433

Add kerberos support for Hive sink

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.5.0.1
    • Fix Version/s: None
    • Component/s: Sinks+Sources

      Description

      Add kerberos authentication support for Hive sink
      FYI: The HCatalog API support for Kerberos is not available in hive 0.13.1
      this should be available in the next hive release.

      1. FLUME-2433.patch
        18 kB
        Roshan Naik
      2. FLUME-2433.v2.patch
        44 kB
        Roshan Naik

        Issue Links

          Activity

          Hide
          Roshan Naik added a comment -

          Johny Rufus It seems like your suggestion is the right way to go.

          My current implementation mimics the previous HDFS kerberos implementation and doesn't use the mod-auth. Right now, I am uploading the rebased patch as it will take me some time to figure out what changes are needed for switching to mod-auth etc. System testing of the new implementation will also take sometime as it requires a secure cluster setup.

          Leave it up to you if you want to commit this in its current state and have the switch it mod-auth in another jira or hold this Jira for the revised implementation. It will take me sometime for me to do that i think.

          Show
          Roshan Naik added a comment - Johny Rufus It seems like your suggestion is the right way to go. My current implementation mimics the previous HDFS kerberos implementation and doesn't use the mod-auth. Right now, I am uploading the rebased patch as it will take me some time to figure out what changes are needed for switching to mod-auth etc. System testing of the new implementation will also take sometime as it requires a secure cluster setup. Leave it up to you if you want to commit this in its current state and have the switch it mod-auth in another jira or hold this Jira for the revised implementation. It will take me sometime for me to do that i think.
          Hide
          Johny Rufus added a comment -

          Sure Roshan Naik, flume-ng-auth module has all the kerberos authentication related code. We probably should leverage the authentication stuff from there, currently we have a model where all the flume components that need authentication (HDFS sink, HBase sink , Thrist src/sink and Dataset sink) are required to use the same credentials [More like authentication creds for the entire Flume agent as a whole]

          Show
          Johny Rufus added a comment - Sure Roshan Naik , flume-ng-auth module has all the kerberos authentication related code. We probably should leverage the authentication stuff from there, currently we have a model where all the flume components that need authentication (HDFS sink, HBase sink , Thrist src/sink and Dataset sink) are required to use the same credentials [More like authentication creds for the entire Flume agent as a whole]
          Hide
          Roshan Naik added a comment -

          This patch probably needs to be rebased. Its actually been in production for over year with the HDP distribution.

          Perhaps Ashish Paliwal or Johny Rufus can help with review and commit once i revise the patch.

          Show
          Roshan Naik added a comment - This patch probably needs to be rebased. Its actually been in production for over year with the HDP distribution. Perhaps Ashish Paliwal or Johny Rufus can help with review and commit once i revise the patch.
          Hide
          Anoop Dawar added a comment -

          Is there a plan to merge this into the mainline?

          Show
          Anoop Dawar added a comment - Is there a plan to merge this into the mainline?
          Hide
          Roshan Naik added a comment -

          uploading patch

          Show
          Roshan Naik added a comment - uploading patch
          Hide
          Roshan Naik added a comment -

          Need to commit FLUME-1734 before committing this one

          Show
          Roshan Naik added a comment - Need to commit FLUME-1734 before committing this one

            People

            • Assignee:
              Roshan Naik
              Reporter:
              Roshan Naik
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development