Flume
  1. Flume
  2. FLUME-1382

Flume adopt message from existing local Scribe

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.3.0
    • Component/s: Sinks+Sources
    • Labels:

      Description

      Currently, we are using Scribe in data ingest system. Central Scribe is hard to maintain and upgrade. Thus, we would like to replace central Scribe with Flume and adopt message from existing and amounts of local Scribe. This can be treated as legacy part.

      We have generated ScribeSource and used with more effective Thrift code without deserializing.

      1. FLUME-1382-doc-2.patch
        2 kB
        Denny Ye
      2. FLUME-1382-3.patch
        59 kB
        Denny Ye

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          2h 31m 1 Denny Ye 19/Jul/12 06:05
          Patch Available Patch Available Resolved Resolved
          27d 1h 55m 1 Juhani Connolly 15/Aug/12 08:00
          Resolved Resolved Reopened Reopened
          1h 54m 1 Juhani Connolly 15/Aug/12 09:54
          Reopened Reopened Resolved Resolved
          16h 58m 1 Juhani Connolly 16/Aug/12 02:53
          Mike Percy made changes -
          Issue Type Improvement [ 4 ] New Feature [ 2 ]
          Xin Yu made changes -
          Link This issue is related to FLUME-1511 [ FLUME-1511 ]
          Hide
          Hari Shreedharan added a comment -

          Xin: Could you please open another jira for this issue?

          Show
          Hari Shreedharan added a comment - Xin: Could you please open another jira for this issue?
          Hide
          Xin Yu added a comment -

          ScribeSource didn't handle correctly for the zero message requests
          from remote scribe server.

          If scribed sends zero message request like:
          sent <0> messages to remote scribe server <dwen:1564>
          log receiver in ScribeSource will return ResultCode.TRY_LATER
          and block whole transfer channel.

          I seggest that return ResultCode.OK when the length of LogEntry's list.

          Show
          Xin Yu added a comment - ScribeSource didn't handle correctly for the zero message requests from remote scribe server. If scribed sends zero message request like: sent <0> messages to remote scribe server <dwen:1564> log receiver in ScribeSource will return ResultCode.TRY_LATER and block whole transfer channel. I seggest that return ResultCode.OK when the length of LogEntry's list.
          Juhani Connolly made changes -
          Resolution Fixed [ 1 ]
          Status Reopened [ 4 ] Resolved [ 5 ]
          Hide
          Juhani Connolly added a comment -

          Ok, it looks like we can't force push to fix the trunk so this isn't getting rolled back.

          I'm going to make another issue for the missing sources from the source tarball

          Show
          Juhani Connolly added a comment - Ok, it looks like we can't force push to fix the trunk so this isn't getting rolled back. I'm going to make another issue for the missing sources from the source tarball
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #282 (See https://builds.apache.org/job/flume-trunk/282/)
          FLUME-1382: Flume adopt message from existing local Scribe (Revision cf50359bd72aa1f5c54a5140009eb3bb7a0d2c02)

          Result = UNSTABLE
          juhanic : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=cf50359bd72aa1f5c54a5140009eb3bb7a0d2c02
          Files :

          • flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ScribeSource.java
          • flume-ng-sources/flume-scribe-source/pom.xml
          • flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java
          • pom.xml
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-sources/pom.xml
          • flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ResultCode.java
          • flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/LogEntry.java
          • flume-ng-dist/pom.xml
          Show
          Hudson added a comment - Integrated in flume-trunk #282 (See https://builds.apache.org/job/flume-trunk/282/ ) FLUME-1382 : Flume adopt message from existing local Scribe (Revision cf50359bd72aa1f5c54a5140009eb3bb7a0d2c02) Result = UNSTABLE juhanic : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git;a=summary&a=commit&h=cf50359bd72aa1f5c54a5140009eb3bb7a0d2c02 Files : flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ScribeSource.java flume-ng-sources/flume-scribe-source/pom.xml flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/Scribe.java pom.xml flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-sources/pom.xml flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/ResultCode.java flume-ng-sources/flume-scribe-source/src/main/java/org/apache/flume/source/scribe/LogEntry.java flume-ng-dist/pom.xml
          Juhani Connolly made changes -
          Status Resolved [ 5 ] Reopened [ 4 ]
          Resolution Fixed [ 1 ]
          Hide
          Juhani Connolly added a comment -

          This is getting rolled back because the git branch I pushed had a couple of merges in it messing the history up, my apologies.

          Furthermore, the source tarball generated is missing the new source files. I added this to the reviewboard

          Show
          Juhani Connolly added a comment - This is getting rolled back because the git branch I pushed had a couple of merges in it messing the history up, my apologies. Furthermore, the source tarball generated is missing the new source files. I added this to the reviewboard
          Juhani Connolly made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Denny Ye made changes -
          Attachment FLUME-1382-doc-2.patch [ 12540847 ]
          Attachment FLUME-1382-3.patch [ 12540848 ]
          Denny Ye made changes -
          Attachment FLUME-1382-doc.patch [ 12539008 ]
          Denny Ye made changes -
          Attachment FLUME-1382.patch [ 12537547 ]
          Hide
          Brock Noland added a comment -

          Neve rmind, I see the RB discussion.

          Show
          Brock Noland added a comment - Neve rmind, I see the RB discussion.
          Hide
          Brock Noland added a comment -

          Hi,

          One question, this means scribe "agents" (or whatever they are called in scribe) would write be able to write to a Flume Source?

          Show
          Brock Noland added a comment - Hi, One question, this means scribe "agents" (or whatever they are called in scribe) would write be able to write to a Flume Source?
          Denny Ye made changes -
          Attachment FLUME-1382-doc.patch [ 12539008 ]
          Denny Ye made changes -
          Attachment FLUME-1382.patch [ 12537547 ]
          Denny Ye made changes -
          Attachment FLUME-1382.txt [ 12537130 ]
          Denny Ye made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Denny Ye added a comment -

          Patch was attached

          Show
          Denny Ye added a comment - Patch was attached
          Denny Ye made changes -
          Field Original Value New Value
          Attachment FLUME-1382.txt [ 12537130 ]
          Hide
          Denny Ye added a comment -

          ScribeSource in legacy sub-folder

          Show
          Denny Ye added a comment - ScribeSource in legacy sub-folder
          Denny Ye created issue -

            People

            • Assignee:
              Denny Ye
              Reporter:
              Denny Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development