[FLUME-1734] Create a Hive Sink based on the new Hive Streaming support - ASF JIRA

Agile Board

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: New Feature
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 1.2.0
Fix Version/s: 1.6.0
Component/s: Sinks+Sources
Labels:
- features

Release Note:

Hide
Committed! Thanks Roshan!

Show
Committed! Thanks Roshan!

Description

Hive 0.13 has introduced Streaming support which is itself transactional in nature and fits well with Flume's transaction model.
Short overview of Hive's Streaming support on which this sink is based can be found here:
http://hive.apache.org/javadocs/r0.13.1/api/hcatalog/streaming/index.html

This jira is for creating a sink that would continuously stream data into Hive tables using the above APIs. The primary goal being that the data streamed by the sink should be instantly queryable (using say Hive or Pig) without requiring additional post-processing steps on behalf of the users. Sink should manage the creation of new partitions periodically if needed.