Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1227

Introduce some sort of SpillableChannel

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.5.0
    • Component/s: Channel
    • Labels:
      None
    • Release Note:
      New Spillable Memory Channel. Functions like Memory channel, but uses disk as overflow if main memory capacity is reached.

      Description

      I would like to introduce new channel that would behave similarly as scribe (https://github.com/facebook/scribe). It would be something between memory and file channel. Input events would be saved directly to the memory (only) and would be served from there. In case that the memory would be full, we would outsource the events to file.

      Let me describe the use case behind this request. We have plenty of frontend servers that are generating events. We want to send all events to just limited number of machines from where we would send the data to HDFS (some sort of staging layer). Reason for this second layer is our need to decouple event aggregation and front end code to separate machines. Using memory channel is fully sufficient as we can survive lost of some portion of the events. However in order to sustain maintenance windows or networking issues we would have to end up with a lot of memory assigned to those "staging" machines. Referenced "scribe" is dealing with this problem by implementing following logic - events are saved in memory similarly as our MemoryChannel. However in case that the memory gets full (because of maintenance, networking issues, ...) it will spill data to disk where they will be sitting until everything start working again.

      I would like to introduce channel that would implement similar logic. It's durability guarantees would be same as MemoryChannel - in case that someone would remove power cord, this channel would lose data. Based on the discussion in FLUME-1201, I would propose to have the implementation completely independent on any other channel internal code.

      Jarcec

        Attachments

        1. SpillableMemory Channel Design.pdf
          252 kB
          Roshan Naik
        2. 1227.patch.1
          74 kB
          Roshan Naik
        3. SpillableMemory Channel Design 2.pdf
          277 kB
          Roshan Naik
        4. FLUME-1227.v2.patch
          16 kB
          Roshan Naik
        5. FLUME-1227.v5.patch
          69 kB
          Roshan Naik
        6. FLUME-1227.v6.patch
          71 kB
          Roshan Naik
        7. FLUME-1227.v7.patch
          72 kB
          Roshan Naik
        8. FLUME-1227.v8.patch
          75 kB
          Roshan Naik
        9. FLUME-1227.v9.patch
          77 kB
          Roshan Naik

          Issue Links

            Activity

              People

              • Assignee:
                roshan_naik Roshan Naik
                Reporter:
                jarcec Jarek Jarcec Cecho
              • Votes:
                1 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: