Flume
  1. Flume
  2. FLUME-1196

Allow different HDFS Sinks within the same agent to write to HDFS as different users

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v1.2.0
    • Component/s: None
    • Labels:
      None

      Description

      The HDFS Event Sink support for Kerberos security affects the logged on user for the entire Agent. Due to this we cannot have two sinks within the same agent that are bound to different principals or users.

        Activity

        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/5163/#review7976
        -----------------------------------------------------------

        I also partially fixed the build issue with HBase sink. Now it compiles on hadoop.profile=23, but the HBase unit tests still fail.

        • Mike

        On 2012-05-18 17:39:31, Mike Percy wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/5163/

        -----------------------------------------------------------

        (Updated 2012-05-18 17:39:31)

        Review request for Flume.

        Summary

        -------

        Support for acting as multiple proxy users from a single, authenticated principal.

        This addresses bug FLUME-1196.

        https://issues.apache.org/jira/browse/FLUME-1196

        Diffs

        -----

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1

        flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac

        flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d

        flume-ng-sinks/flume-ng-hbase-sink/pom.xml ed2e44a

        pom.xml 7a97c34

        Diff: https://reviews.apache.org/r/5163/diff

        Testing

        -------

        Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/#review7976 ----------------------------------------------------------- I also partially fixed the build issue with HBase sink. Now it compiles on hadoop.profile=23, but the HBase unit tests still fail. Mike On 2012-05-18 17:39:31, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/ ----------------------------------------------------------- (Updated 2012-05-18 17:39:31) Review request for Flume. Summary ------- Support for acting as multiple proxy users from a single, authenticated principal. This addresses bug FLUME-1196 . https://issues.apache.org/jira/browse/FLUME-1196 Diffs ----- flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1 flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d flume-ng-sinks/flume-ng-hbase-sink/pom.xml ed2e44a pom.xml 7a97c34 Diff: https://reviews.apache.org/r/5163/diff Testing ------- Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/5163/
        -----------------------------------------------------------

        Review request for Flume.

        Summary
        -------

        Support for acting as multiple proxy users from a single, authenticated principal.

        This addresses bug FLUME-1196.
        https://issues.apache.org/jira/browse/FLUME-1196

        Diffs


        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1
        flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac
        flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d
        flume-ng-sinks/flume-ng-hbase-sink/pom.xml ed2e44a
        pom.xml 7a97c34

        Diff: https://reviews.apache.org/r/5163/diff

        Testing
        -------

        Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/ ----------------------------------------------------------- Review request for Flume. Summary ------- Support for acting as multiple proxy users from a single, authenticated principal. This addresses bug FLUME-1196 . https://issues.apache.org/jira/browse/FLUME-1196 Diffs flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1 flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d flume-ng-sinks/flume-ng-hbase-sink/pom.xml ed2e44a pom.xml 7a97c34 Diff: https://reviews.apache.org/r/5163/diff Testing ------- Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/5163/#review7977
        -----------------------------------------------------------

        Mike - It is better not to provide HbaseSink compiled against hadoop-23, since it will fail at run time(on hadoop-23), due to hadoop's compile only compatibility. So as part of FLUME-1208, I have removed Hbase sink compile itself in hadoop-23 profile. Please take a look at HBASE-6020. Until that is resolved, there is no point in building the Hbase sink against hadoop-23.

        • Hari

        On 2012-05-18 17:39:31, Mike Percy wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/5163/

        -----------------------------------------------------------

        (Updated 2012-05-18 17:39:31)

        Review request for Flume.

        Summary

        -------

        Support for acting as multiple proxy users from a single, authenticated principal.

        This addresses bug FLUME-1196.

        https://issues.apache.org/jira/browse/FLUME-1196

        Diffs

        -----

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1

        flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac

        flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d

        flume-ng-sinks/flume-ng-hbase-sink/pom.xml ed2e44a

        pom.xml 7a97c34

        Diff: https://reviews.apache.org/r/5163/diff

        Testing

        -------

        Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass.

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/#review7977 ----------------------------------------------------------- Mike - It is better not to provide HbaseSink compiled against hadoop-23, since it will fail at run time(on hadoop-23), due to hadoop's compile only compatibility. So as part of FLUME-1208 , I have removed Hbase sink compile itself in hadoop-23 profile. Please take a look at HBASE-6020 . Until that is resolved, there is no point in building the Hbase sink against hadoop-23. Hari On 2012-05-18 17:39:31, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/ ----------------------------------------------------------- (Updated 2012-05-18 17:39:31) Review request for Flume. Summary ------- Support for acting as multiple proxy users from a single, authenticated principal. This addresses bug FLUME-1196 . https://issues.apache.org/jira/browse/FLUME-1196 Diffs ----- flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1 flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d flume-ng-sinks/flume-ng-hbase-sink/pom.xml ed2e44a pom.xml 7a97c34 Diff: https://reviews.apache.org/r/5163/diff Testing ------- Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/5163/
        -----------------------------------------------------------

        (Updated 2012-05-18 18:55:41.519231)

        Review request for Flume.

        Summary
        -------

        Support for acting as multiple proxy users from a single, authenticated principal.

        This addresses bug FLUME-1196.
        https://issues.apache.org/jira/browse/FLUME-1196

        Diffs


        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1
        flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac
        flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d
        flume-ng-sinks/flume-ng-hbase-sink/pom.xml ed2e44a
        pom.xml 7a97c34

        Diff: https://reviews.apache.org/r/5163/diff

        Testing (updated)
        -------

        Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Also, a non-trivial amount of manual testing (no Kerberos, Kerb only, Kerb+proxy, combination of Kerb & Kerb+proxy in same agent)

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/ ----------------------------------------------------------- (Updated 2012-05-18 18:55:41.519231) Review request for Flume. Summary ------- Support for acting as multiple proxy users from a single, authenticated principal. This addresses bug FLUME-1196 . https://issues.apache.org/jira/browse/FLUME-1196 Diffs flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1 flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d flume-ng-sinks/flume-ng-hbase-sink/pom.xml ed2e44a pom.xml 7a97c34 Diff: https://reviews.apache.org/r/5163/diff Testing (updated) ------- Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Also, a non-trivial amount of manual testing (no Kerberos, Kerb only, Kerb+proxy, combination of Kerb & Kerb+proxy in same agent) Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2012-05-18 18:31:53, Hari Shreedharan wrote:

        > Mike - It is better not to provide HbaseSink compiled against hadoop-23, since it will fail at run time(on hadoop-23), due to hadoop's compile only compatibility. So as part of FLUME-1208, I have removed Hbase sink compile itself in hadoop-23 profile. Please take a look at HBASE-6020. Until that is resolved, there is no point in building the Hbase sink against hadoop-23.

        I see. Yeah I just wanted the build to compile. Thanks for looking at this Hari! I'll update the patch.

        • Mike

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/5163/#review7977
        -----------------------------------------------------------

        On 2012-05-18 18:55:41, Mike Percy wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/5163/

        -----------------------------------------------------------

        (Updated 2012-05-18 18:55:41)

        Review request for Flume.

        Summary

        -------

        Support for acting as multiple proxy users from a single, authenticated principal.

        This addresses bug FLUME-1196.

        https://issues.apache.org/jira/browse/FLUME-1196

        Diffs

        -----

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1

        flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac

        flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d

        flume-ng-sinks/flume-ng-hbase-sink/pom.xml ed2e44a

        pom.xml 7a97c34

        Diff: https://reviews.apache.org/r/5163/diff

        Testing

        -------

        Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Also, a non-trivial amount of manual testing (no Kerberos, Kerb only, Kerb+proxy, combination of Kerb & Kerb+proxy in same agent)

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - On 2012-05-18 18:31:53, Hari Shreedharan wrote: > Mike - It is better not to provide HbaseSink compiled against hadoop-23, since it will fail at run time(on hadoop-23), due to hadoop's compile only compatibility. So as part of FLUME-1208 , I have removed Hbase sink compile itself in hadoop-23 profile. Please take a look at HBASE-6020 . Until that is resolved, there is no point in building the Hbase sink against hadoop-23. I see. Yeah I just wanted the build to compile. Thanks for looking at this Hari! I'll update the patch. Mike ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/#review7977 ----------------------------------------------------------- On 2012-05-18 18:55:41, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/ ----------------------------------------------------------- (Updated 2012-05-18 18:55:41) Review request for Flume. Summary ------- Support for acting as multiple proxy users from a single, authenticated principal. This addresses bug FLUME-1196 . https://issues.apache.org/jira/browse/FLUME-1196 Diffs ----- flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1 flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d flume-ng-sinks/flume-ng-hbase-sink/pom.xml ed2e44a pom.xml 7a97c34 Diff: https://reviews.apache.org/r/5163/diff Testing ------- Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Also, a non-trivial amount of manual testing (no Kerberos, Kerb only, Kerb+proxy, combination of Kerb & Kerb+proxy in same agent) Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/5163/
        -----------------------------------------------------------

        (Updated 2012-05-18 21:51:36.018974)

        Review request for Flume.

        Changes
        -------

        Updated diff to remove hbase-related build changes.

        Summary
        -------

        Support for acting as multiple proxy users from a single, authenticated principal.

        This addresses bug FLUME-1196.
        https://issues.apache.org/jira/browse/FLUME-1196

        Diffs (updated)


        flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d
        pom.xml 7a97c34

        Diff: https://reviews.apache.org/r/5163/diff

        Testing
        -------

        Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Also, a non-trivial amount of manual testing (no Kerberos, Kerb only, Kerb+proxy, combination of Kerb & Kerb+proxy in same agent)

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/ ----------------------------------------------------------- (Updated 2012-05-18 21:51:36.018974) Review request for Flume. Changes ------- Updated diff to remove hbase-related build changes. Summary ------- Support for acting as multiple proxy users from a single, authenticated principal. This addresses bug FLUME-1196 . https://issues.apache.org/jira/browse/FLUME-1196 Diffs (updated) flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d pom.xml 7a97c34 Diff: https://reviews.apache.org/r/5163/diff Testing ------- Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Also, a non-trivial amount of manual testing (no Kerberos, Kerb only, Kerb+proxy, combination of Kerb & Kerb+proxy in same agent) Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/5163/#review7984
        -----------------------------------------------------------

        Ship it!

        +1

        Thanks for the patch Mike, changes look good!

        • Arvind

        On 2012-05-18 21:51:36, Mike Percy wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/5163/

        -----------------------------------------------------------

        (Updated 2012-05-18 21:51:36)

        Review request for Flume.

        Summary

        -------

        Support for acting as multiple proxy users from a single, authenticated principal.

        This addresses bug FLUME-1196.

        https://issues.apache.org/jira/browse/FLUME-1196

        Diffs

        -----

        flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1

        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION

        flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d

        pom.xml 7a97c34

        Diff: https://reviews.apache.org/r/5163/diff

        Testing

        -------

        Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Also, a non-trivial amount of manual testing (no Kerberos, Kerb only, Kerb+proxy, combination of Kerb & Kerb+proxy in same agent)

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/#review7984 ----------------------------------------------------------- Ship it! +1 Thanks for the patch Mike, changes look good! Arvind On 2012-05-18 21:51:36, Mike Percy wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/ ----------------------------------------------------------- (Updated 2012-05-18 21:51:36) Review request for Flume. Summary ------- Support for acting as multiple proxy users from a single, authenticated principal. This addresses bug FLUME-1196 . https://issues.apache.org/jira/browse/FLUME-1196 Diffs ----- flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d pom.xml 7a97c34 Diff: https://reviews.apache.org/r/5163/diff Testing ------- Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Also, a non-trivial amount of manual testing (no Kerberos, Kerb only, Kerb+proxy, combination of Kerb & Kerb+proxy in same agent) Thanks, Mike
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/5163/
        -----------------------------------------------------------

        (Updated 2012-05-19 00:25:01.910647)

        Review request for Flume.

        Changes
        -------

        I don't think there is a difference in these patches (except for git hashes related to how I rebased). But I realized I attached just slightly different patches to this review & the JIRA. This one is a copy of the one on the JIRA.

        As before, all unit tests pass.

        Summary
        -------

        Support for acting as multiple proxy users from a single, authenticated principal.

        This addresses bug FLUME-1196.
        https://issues.apache.org/jira/browse/FLUME-1196

        Diffs (updated)


        flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1
        flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION
        flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d
        pom.xml 7a97c34

        Diff: https://reviews.apache.org/r/5163/diff

        Testing
        -------

        Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Also, a non-trivial amount of manual testing (no Kerberos, Kerb only, Kerb+proxy, combination of Kerb & Kerb+proxy in same agent)

        Thanks,

        Mike

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/5163/ ----------------------------------------------------------- (Updated 2012-05-19 00:25:01.910647) Review request for Flume. Changes ------- I don't think there is a difference in these patches (except for git hashes related to how I rebased). But I realized I attached just slightly different patches to this review & the JIRA. This one is a copy of the one on the JIRA. As before, all unit tests pass. Summary ------- Support for acting as multiple proxy users from a single, authenticated principal. This addresses bug FLUME-1196 . https://issues.apache.org/jira/browse/FLUME-1196 Diffs (updated) flume-ng-sinks/flume-hdfs-sink/pom.xml 3277a00 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java 8ce2bac flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java d7963b1 flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java PRE-CREATION flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties 361158d pom.xml 7a97c34 Diff: https://reviews.apache.org/r/5163/diff Testing ------- Added unit test and small framework for testing using MiniDFSCluster. Existing tests pass. Also, a non-trivial amount of manual testing (no Kerberos, Kerb only, Kerb+proxy, combination of Kerb & Kerb+proxy in same agent) Thanks, Mike
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Mike!

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Mike!
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #208 (See https://builds.apache.org/job/flume-trunk/208/)
        FLUME-1196. Support for multiple principals within an agent.

        (Mike Percy via Arvind Prabhakar) (Revision 1340323)

        Result = SUCCESS
        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1340323
        Files :

        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/pom.xml
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java
        • /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties
        • /incubator/flume/trunk/pom.xml
        Show
        Hudson added a comment - Integrated in flume-trunk #208 (See https://builds.apache.org/job/flume-trunk/208/ ) FLUME-1196 . Support for multiple principals within an agent. (Mike Percy via Arvind Prabhakar) (Revision 1340323) Result = SUCCESS arvind : http://svn.apache.org/viewvc/?view=rev&rev=1340323 Files : /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/pom.xml /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSEventSink.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/HDFSFormatterFactory.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/main/java/org/apache/flume/sink/hdfs/KerberosUser.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/java/org/apache/flume/sink/hdfs/TestHDFSEventSinkOnMiniCluster.java /incubator/flume/trunk/flume-ng-sinks/flume-hdfs-sink/src/test/resources/log4j.properties /incubator/flume/trunk/pom.xml

          People

          • Assignee:
            Mike Percy
            Reporter:
            Arvind Prabhakar
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development