Flume
  1. Flume
  2. FLUME-2373

Enable configuration to switch thrift source serialization protocol

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      The serialization protocol for the thrift source is hard coded to be "compact". This change allows you to set it to "binary" via configuration.

      This change is fully backwards compatible with current configuration that people would have, since it defaults to "compact".

      Note:

      • It doesn't go all the way to expose all different serialization options.
      • I'm a n00b at this, so let me know if I'm missing something please
      1. FLUME-2373-1.patch
        14 kB
        Stefan Krawczyk

        Activity

        Hide
        Stefan Krawczyk added a comment -

        Here's the patch

        Show
        Stefan Krawczyk added a comment - Here's the patch
        Hide
        Hari Shreedharan added a comment -

        Hi Stefan,

        This generally looks good, but we'd want to support these options in the Thrift RPC Client and Thrift Sink as well.

        Show
        Hari Shreedharan added a comment - Hi Stefan, This generally looks good, but we'd want to support these options in the Thrift RPC Client and Thrift Sink as well.
        Hide
        Stefan Krawczyk added a comment -

        Okay, so you'd want that in a single patch?

        Show
        Stefan Krawczyk added a comment - Okay, so you'd want that in a single patch?
        Hide
        Hari Shreedharan added a comment -

        Yep, that would be the best thing to do.

        Show
        Hari Shreedharan added a comment - Yep, that would be the best thing to do.
        Hide
        Stefan Krawczyk added a comment -

        New patch – did I miss any spots? Seems like the Thrift RPC client and the thrift sink are one and the same?

        Show
        Stefan Krawczyk added a comment - New patch – did I miss any spots? Seems like the Thrift RPC client and the thrift sink are one and the same?
        Hide
        Hari Shreedharan added a comment -

        +1. This looks good. I am going to run tests and commit this. Do you mind filing a jira to add the new parameter to the docs?

        Show
        Hari Shreedharan added a comment - +1. This looks good. I am going to run tests and commit this. Do you mind filing a jira to add the new parameter to the docs?
        Hide
        Stefan Krawczyk added a comment -

        Done. FLUME-2577

        Show
        Stefan Krawczyk added a comment - Done. FLUME-2577
        Hide
        ASF subversion and git services added a comment -

        Commit 34e9bda312506a118fad87fcbdecc48bf3918c95 in flume's branch refs/heads/trunk from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=34e9bda ]

        FLUME-2373. Support TBinaryProtocol in Thrift RPC.

        (Stefan Krawczyk via Hari)

        Show
        ASF subversion and git services added a comment - Commit 34e9bda312506a118fad87fcbdecc48bf3918c95 in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=34e9bda ] FLUME-2373 . Support TBinaryProtocol in Thrift RPC. (Stefan Krawczyk via Hari)
        Hide
        ASF subversion and git services added a comment -

        Commit 49101731c35ed5674b668993dc00a7635c2eee20 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
        [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4910173 ]

        FLUME-2373. Support TBinaryProtocol in Thrift RPC.

        (Stefan Krawczyk via Hari)

        Show
        ASF subversion and git services added a comment - Commit 49101731c35ed5674b668993dc00a7635c2eee20 in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4910173 ] FLUME-2373 . Support TBinaryProtocol in Thrift RPC. (Stefan Krawczyk via Hari)
        Hide
        Hari Shreedharan added a comment -

        Committed! Thanks Stefan!

        Show
        Hari Shreedharan added a comment - Committed! Thanks Stefan!
        Hide
        Hudson added a comment -

        UNSTABLE: Integrated in flume-trunk #704 (See https://builds.apache.org/job/flume-trunk/704/)
        FLUME-2373. Support TBinaryProtocol in Thrift RPC. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=34e9bda312506a118fad87fcbdecc48bf3918c95)

        • flume-ng-core/src/test/java/org/apache/flume/sink/TestThriftSink.java
        • flume-ng-sdk/src/test/java/org/apache/flume/api/TestThriftRpcClient.java
        • flume-ng-core/src/main/java/org/apache/flume/sink/ThriftSink.java
        • flume-ng-sdk/src/test/java/org/apache/flume/api/ThriftTestingSource.java
        • flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java
        • flume-ng-sdk/src/main/java/org/apache/flume/api/ThriftRpcClient.java
        Show
        Hudson added a comment - UNSTABLE: Integrated in flume-trunk #704 (See https://builds.apache.org/job/flume-trunk/704/ ) FLUME-2373 . Support TBinaryProtocol in Thrift RPC. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=34e9bda312506a118fad87fcbdecc48bf3918c95 ) flume-ng-core/src/test/java/org/apache/flume/sink/TestThriftSink.java flume-ng-sdk/src/test/java/org/apache/flume/api/TestThriftRpcClient.java flume-ng-core/src/main/java/org/apache/flume/sink/ThriftSink.java flume-ng-sdk/src/test/java/org/apache/flume/api/ThriftTestingSource.java flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java flume-ng-sdk/src/main/java/org/apache/flume/api/ThriftRpcClient.java
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Flume-trunk-hbase-98 #62 (See https://builds.apache.org/job/Flume-trunk-hbase-98/62/)
        FLUME-2373. Support TBinaryProtocol in Thrift RPC. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=34e9bda312506a118fad87fcbdecc48bf3918c95)

        • flume-ng-sdk/src/main/java/org/apache/flume/api/ThriftRpcClient.java
        • flume-ng-core/src/main/java/org/apache/flume/sink/ThriftSink.java
        • flume-ng-sdk/src/test/java/org/apache/flume/api/ThriftTestingSource.java
        • flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java
        • flume-ng-sdk/src/test/java/org/apache/flume/api/TestThriftRpcClient.java
        • flume-ng-core/src/test/java/org/apache/flume/sink/TestThriftSink.java
        Show
        Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-98 #62 (See https://builds.apache.org/job/Flume-trunk-hbase-98/62/ ) FLUME-2373 . Support TBinaryProtocol in Thrift RPC. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=34e9bda312506a118fad87fcbdecc48bf3918c95 ) flume-ng-sdk/src/main/java/org/apache/flume/api/ThriftRpcClient.java flume-ng-core/src/main/java/org/apache/flume/sink/ThriftSink.java flume-ng-sdk/src/test/java/org/apache/flume/api/ThriftTestingSource.java flume-ng-core/src/main/java/org/apache/flume/source/ThriftSource.java flume-ng-sdk/src/test/java/org/apache/flume/api/TestThriftRpcClient.java flume-ng-core/src/test/java/org/apache/flume/sink/TestThriftSink.java

          People

          • Assignee:
            Stefan Krawczyk
            Reporter:
            Stefan Krawczyk
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development