Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1734

Create a Hive Sink based on the new Hive Streaming support

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.6.0
    • Component/s: Sinks+Sources
    • Labels:
    • Release Note:
      Hide
      Committed! Thanks Roshan!
      Show
      Committed! Thanks Roshan!

      Description

      Hive 0.13 has introduced Streaming support which is itself transactional in nature and fits well with Flume's transaction model.
      Short overview of Hive's Streaming support on which this sink is based can be found here:
      http://hive.apache.org/javadocs/r0.13.1/api/hcatalog/streaming/index.html

      This jira is for creating a sink that would continuously stream data into Hive tables using the above APIs. The primary goal being that the data streamed by the sink should be instantly queryable (using say Hive or Pig) without requiring additional post-processing steps on behalf of the users. Sink should manage the creation of new partitions periodically if needed.

        Attachments

        1. FLUME-1734.v10b.patch
          109 kB
          Roshan Naik
        2. FLUME-1734.v10.patch
          109 kB
          Roshan Naik
        3. FLUME-1734.v9.patch
          109 kB
          Roshan Naik
        4. FLUME-1734.v8.patch
          108 kB
          Roshan Naik
        5. FLUME-1734.v7.patch
          24 kB
          Roshan Naik
        6. FLUME-1734.v6.patch
          108 kB
          Roshan Naik
        7. FLUME-1734.v5.patch
          110 kB
          Roshan Naik
        8. FLUME-1734.v4.patch
          109 kB
          Roshan Naik
        9. FLUME-1734.v2.patch
          107 kB
          Roshan Naik
        10. FLUME-1734.v1.patch
          103 kB
          Roshan Naik
        11. FLUME-1734.draft.2.patch
          45 kB
          Roshan Naik
        12. FLUME-1734.draft.1.patch
          0.5 kB
          Roshan Naik

          Issue Links

            Activity

              People

              • Assignee:
                roshan_naik Roshan Naik
                Reporter:
                roshan_naik Roshan Naik
              • Votes:
                0 Vote for this issue
                Watchers:
                12 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: