Flume
  1. Flume
  2. FLUME-214

Flume should be able to compile/run against non-kerberos hdfs

    Details

      Description

      The current kerberos patch lives in a separate hdfs-security branch because it depends on new security-enbled hdfs apis to bulid/run.

        Issue Links

          Activity

          Hide
          Jonathan Hsieh added a comment -

          FLUME-197 introduced a dependency on org.apache.hadoop.security.SecurityUtil and .UserGroupInformation.

          UGI exists in hadoop 0.18.3 and 0.20.x+
          Some of the calls on UGI did not exist in 0.18.3 and does exist in 0.20.x.
          SU does not exist in hadoop 0.18.3 but does on 0.20.x+

          We should be able to justload UGI in all versions, but are going to use reflection to get SU, get the methods on UGI and SU, and to invoke the methods necessary for Kerberos authentication.

          Show
          Jonathan Hsieh added a comment - FLUME-197 introduced a dependency on org.apache.hadoop.security.SecurityUtil and .UserGroupInformation. UGI exists in hadoop 0.18.3 and 0.20.x+ Some of the calls on UGI did not exist in 0.18.3 and does exist in 0.20.x. SU does not exist in hadoop 0.18.3 but does on 0.20.x+ We should be able to justload UGI in all versions, but are going to use reflection to get SU, get the methods on UGI and SU, and to invoke the methods necessary for Kerberos authentication.
          Hide
          Jonathan Hsieh added a comment -

          Manual testing procedure:
          bin/flume node -1 -s -n kerb -c 'kerb:asciisynth(10)|collectorSink("hdfs://kerbnn/user/jon/reflection","test");'

          kinit flume

          hadoop dfs -ls /user/jon/reflection/* | wc

          10 lines as expected.

          Show
          Jonathan Hsieh added a comment - Manual testing procedure: bin/flume node -1 -s -n kerb -c 'kerb:asciisynth(10)|collectorSink("hdfs://kerbnn/user/jon/reflection","test");' kinit flume hadoop dfs -ls /user/jon/reflection/* | wc 10 lines as expected.
          Hide
          Jonathan Hsieh added a comment -

          Works in kerberized hadoop with proper keytabs, also works in 0.20.2+230 build without kerberos.

          Show
          Jonathan Hsieh added a comment - Works in kerberized hadoop with proper keytabs, also works in 0.20.2+230 build without kerberos.
          Show
          Jonathan Hsieh added a comment - https://review.cloudera.org/r/901/
          Hide
          Jonathan Hsieh added a comment -

          Attached post-review updated patch

          Show
          Jonathan Hsieh added a comment - Attached post-review updated patch
          Hide
          Jonathan Hsieh added a comment -

          Pushed to hdfs-security branch.

          Show
          Jonathan Hsieh added a comment - Pushed to hdfs-security branch.
          Hide
          Jonathan Hsieh added a comment -

          Committed to trunk now.

          Show
          Jonathan Hsieh added a comment - Committed to trunk now.
          Hide
          Jonathan Hsieh added a comment -

          Closing released issues.

          Show
          Jonathan Hsieh added a comment - Closing released issues.

            People

            • Assignee:
              Jonathan Hsieh
              Reporter:
              Jonathan Hsieh
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development