Flume
  1. Flume
  2. FLUME-1734

Create a Hive Sink based on the new Hive Streaming support

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.6.0
    • Component/s: Sinks+Sources
    • Labels:
    • Release Note:
      Hide
      Committed! Thanks Roshan!
      Show
      Committed! Thanks Roshan!

      Description

      Hive 0.13 has introduced Streaming support which is itself transactional in nature and fits well with Flume's transaction model.
      Short overview of Hive's Streaming support on which this sink is based can be found here:
      http://hive.apache.org/javadocs/r0.13.1/api/hcatalog/streaming/index.html

      This jira is for creating a sink that would continuously stream data into Hive tables using the above APIs. The primary goal being that the data streamed by the sink should be instantly queryable (using say Hive or Pig) without requiring additional post-processing steps on behalf of the users. Sink should manage the creation of new partitions periodically if needed.

      1. FLUME-1734.draft.1.patch
        0.5 kB
        Roshan Naik
      2. FLUME-1734.draft.2.patch
        45 kB
        Roshan Naik
      3. FLUME-1734.v1.patch
        103 kB
        Roshan Naik
      4. FLUME-1734.v2.patch
        107 kB
        Roshan Naik
      5. FLUME-1734.v4.patch
        109 kB
        Roshan Naik
      6. FLUME-1734.v5.patch
        110 kB
        Roshan Naik
      7. FLUME-1734.v6.patch
        108 kB
        Roshan Naik
      8. FLUME-1734.v7.patch
        24 kB
        Roshan Naik
      9. FLUME-1734.v8.patch
        108 kB
        Roshan Naik
      10. FLUME-1734.v9.patch
        109 kB
        Roshan Naik
      11. FLUME-1734.v10.patch
        109 kB
        Roshan Naik
      12. FLUME-1734.v10b.patch
        109 kB
        Roshan Naik

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Roshan Naik
              Reporter:
              Roshan Naik
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development