Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1734

Create a Hive Sink based on the new Hive Streaming support

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.2.0
    • 1.6.0
    • Sinks+Sources
    • Hide
      Committed! Thanks Roshan!
      Show
      Committed! Thanks Roshan!

    Description

      Hive 0.13 has introduced Streaming support which is itself transactional in nature and fits well with Flume's transaction model.
      Short overview of Hive's Streaming support on which this sink is based can be found here:
      http://hive.apache.org/javadocs/r0.13.1/api/hcatalog/streaming/index.html

      This jira is for creating a sink that would continuously stream data into Hive tables using the above APIs. The primary goal being that the data streamed by the sink should be instantly queryable (using say Hive or Pig) without requiring additional post-processing steps on behalf of the users. Sink should manage the creation of new partitions periodically if needed.

      Attachments

        1. FLUME-1734.v9.patch
          109 kB
          Roshan Naik
        2. FLUME-1734.v8.patch
          108 kB
          Roshan Naik
        3. FLUME-1734.v7.patch
          24 kB
          Roshan Naik
        4. FLUME-1734.v6.patch
          108 kB
          Roshan Naik
        5. FLUME-1734.v5.patch
          110 kB
          Roshan Naik
        6. FLUME-1734.v4.patch
          109 kB
          Roshan Naik
        7. FLUME-1734.v2.patch
          107 kB
          Roshan Naik
        8. FLUME-1734.v10b.patch
          109 kB
          Roshan Naik
        9. FLUME-1734.v10.patch
          109 kB
          Roshan Naik
        10. FLUME-1734.v1.patch
          103 kB
          Roshan Naik
        11. FLUME-1734.draft.2.patch
          45 kB
          Roshan Naik
        12. FLUME-1734.draft.1.patch
          0.5 kB
          Roshan Naik

        Issue Links

          Activity

            People

              roshan_naik Roshan Naik
              roshan_naik Roshan Naik
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: