Flume
  1. Flume
  2. FLUME-1382

Flume adopt message from existing local Scribe

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.3.0
    • Component/s: Sinks+Sources
    • Labels:

      Description

      Currently, we are using Scribe in data ingest system. Central Scribe is hard to maintain and upgrade. Thus, we would like to replace central Scribe with Flume and adopt message from existing and amounts of local Scribe. This can be treated as legacy part.

      We have generated ScribeSource and used with more effective Thrift code without deserializing.

      1. FLUME-1382-3.patch
        59 kB
        Denny Ye
      2. FLUME-1382-doc-2.patch
        2 kB
        Denny Ye

        Issue Links

          Activity

          Hide
          Denny Ye added a comment -

          ScribeSource in legacy sub-folder

          Show
          Denny Ye added a comment - ScribeSource in legacy sub-folder
          Hide
          Denny Ye added a comment -

          Patch was attached

          Show
          Denny Ye added a comment - Patch was attached
          Hide
          Brock Noland added a comment -

          Hi,

          One question, this means scribe "agents" (or whatever they are called in scribe) would write be able to write to a Flume Source?

          Show
          Brock Noland added a comment - Hi, One question, this means scribe "agents" (or whatever they are called in scribe) would write be able to write to a Flume Source?
          Hide
          Brock Noland added a comment -

          Neve rmind, I see the RB discussion.

          Show
          Brock Noland added a comment - Neve rmind, I see the RB discussion.
          Hide
          Juhani Connolly added a comment -

          This is getting rolled back because the git branch I pushed had a couple of merges in it messing the history up, my apologies.

          Furthermore, the source tarball generated is missing the new source files. I added this to the reviewboard

          Show
          Juhani Connolly added a comment - This is getting rolled back because the git branch I pushed had a couple of merges in it messing the history up, my apologies. Furthermore, the source tarball generated is missing the new source files. I added this to the reviewboard
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #282 (See https://builds.apache.org/job/flume-trunk/282/)
          FLUME-1382: Flume adopt message from existing local Scribe (Revision cf50359bd72aa1f5c54a5140009eb3bb7a0d2c02)

          Result = UNSTABLE
          juhanic : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=cf50359bd72aa1f5c54a5140009eb3bb7a0d2c02
          Files :

          • flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ScribeSource.java
          • flume-ng-sources/flume-scribe-source/pom.xml
          • flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java
          • pom.xml
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-sources/pom.xml
          • flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ResultCode.java
          • flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/LogEntry.java
          • flume-ng-dist/pom.xml
          Show
          Hudson added a comment - Integrated in flume-trunk #282 (See https://builds.apache.org/job/flume-trunk/282/ ) FLUME-1382 : Flume adopt message from existing local Scribe (Revision cf50359bd72aa1f5c54a5140009eb3bb7a0d2c02) Result = UNSTABLE juhanic : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=cf50359bd72aa1f5c54a5140009eb3bb7a0d2c02 Files : flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ScribeSource.java flume-ng-sources/flume-scribe-source/pom.xml flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java pom.xml flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sources/pom.xml flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ResultCode.java flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/LogEntry.java flume-ng-dist/pom.xml
          Hide
          Juhani Connolly added a comment -

          Ok, it looks like we can't force push to fix the trunk so this isn't getting rolled back.

          I'm going to make another issue for the missing sources from the source tarball

          Show
          Juhani Connolly added a comment - Ok, it looks like we can't force push to fix the trunk so this isn't getting rolled back. I'm going to make another issue for the missing sources from the source tarball
          Hide
          Xin Yu added a comment -

          ScribeSource didn't handle correctly for the zero message requests
          from remote scribe server.

          If scribed sends zero message request like:
          sent <0> messages to remote scribe server <dwen:1564>
          log receiver in ScribeSource will return ResultCode.TRY_LATER
          and block whole transfer channel.

          I seggest that return ResultCode.OK when the length of LogEntry's list.

          Show
          Xin Yu added a comment - ScribeSource didn't handle correctly for the zero message requests from remote scribe server. If scribed sends zero message request like: sent <0> messages to remote scribe server <dwen:1564> log receiver in ScribeSource will return ResultCode.TRY_LATER and block whole transfer channel. I seggest that return ResultCode.OK when the length of LogEntry's list.
          Hide
          Hari Shreedharan added a comment -

          Xin: Could you please open another jira for this issue?

          Show
          Hari Shreedharan added a comment - Xin: Could you please open another jira for this issue?

            People

            • Assignee:
              Denny Ye
              Reporter:
              Denny Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development