Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.1.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      The HDFS sink's directory structure and bucketing is mostly sufficient for Hive tables. The bucketing could result into new directories getting created. This requrires altering Hive table to add a new partition for the directory.
      This hive sink is basically to handle new directories getting added to a target table. The functionality discussed in FLUME-990 will be tracked as part of that ticket.

        Activity

        Hide
        E. Sammer added a comment -

        In 0.9.x there was the notion of a Hive sink as well. Since there's so much repeated code between what the HDFS sink does and what the Hive sink would do - and because I can imagine this being common - why not just create events and observers and have orthogonal plugins that can listen and react to events. For instance, the HDFSSink may support generating the events OnFileOpen, OnFileClose, OnAppend, etc.

        I wrote a general purpose library to handle things like this at one point (we don't have to use it - it's just an example of what I'm talking about): https://github.com/esammer/observer

        Show
        E. Sammer added a comment - In 0.9.x there was the notion of a Hive sink as well. Since there's so much repeated code between what the HDFS sink does and what the Hive sink would do - and because I can imagine this being common - why not just create events and observers and have orthogonal plugins that can listen and react to events. For instance, the HDFSSink may support generating the events OnFileOpen, OnFileClose, OnAppend, etc. I wrote a general purpose library to handle things like this at one point (we don't have to use it - it's just an example of what I'm talking about): https://github.com/esammer/observer
        Hide
        Alexander Alten-Lorenz added a comment -

        FLUME-990 duplicate

        Show
        Alexander Alten-Lorenz added a comment - FLUME-990 duplicate

          People

          • Assignee:
            Prasad Mujumdar
            Reporter:
            Prasad Mujumdar
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development