[FLUME-3262] Improve Scribe Source With More Options

• Type: Improvement
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: zhenzhao wang
• Created: 27/Jul/18 21:02
• Updated: 01/May/20 20:08

Description

We're a heavy scribe source user with tremendous traffic. We found there're limit configuration options in original scribe source. So we add more options including:

1. ThriftServerType, user choose one of {THshaServer, TThreadedSelectorServer, TThreadPoolServer}

   base their own traffic pattern. More details on thrift server could be found on Apache Thrift project.

2. maxThriftFrameSizeBytes, the default max thrift frame size is 16M, it should be good most times, however there might some cases that the thrift size is bigger than 16M. The thrift server will just throw exception without too many valid info when a msg is bigger than the threshold. Some user might prefer receive, print out some msg before discard it, or some user just have event size bigger than 16M
3. Max read buffer size. The buffer size thrift used to handle requests, increase the buffer will benefit throughput under heavy load.

Issue Links

• links to: GitHub Pull Request #219