[FLUME-1734] Create a Hive Sink based on the new Hive Streaming support

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0
• Fix Version/s: 1.6.0
• Component/s: Sinks+Sources
• Labels: features

Release Note

Committed! Thanks Roshan!

• Assignee: Roshan Naik
• Reporter: Roshan Naik
• Created: 20/Nov/12 20:07
• Updated: 17/Apr/15 00:19
• Resolved: 10/Feb/15 07:17

Description

Hive 0.13 has introduced Streaming support which is itself transactional in nature and fits well with Flume's transaction model.
Short overview of Hive's Streaming support on which this sink is based can be found here:
http://hive.apache.org/javadocs/r0.13.1/api/hcatalog/streaming/index.html

This jira is for creating a sink that would continuously stream data into Hive tables using the above APIs. The primary goal being that the data streamed by the sink should be instantly queryable (using say Hive or Pig) without requiring additional post-processing steps on behalf of the users. Sink should manage the creation of new partitions periodically if needed.

Issue Links

• blocks: FLUME-2433 Add kerberos support for Hive sink
• depends upon: HIVE-5687 Streaming support in Hive
• links to: Review Board Link