Flume
  1. Flume
  2. FLUME-1150

Need a way to specify bucket in HDFSEventSink from client

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      We need a way to specify the bucket when event is streamed from client.

      For instance, if i have 5 different log categories (like category [1-5]), if i use single source->channel->hdfsSink then all the logs goes to one specified directory in HDFS (/logs/<destination>) but we need something like

      /logs/category1
      /category2
      /category3
      /category4
      /category5

      I can use multiplexing (1 source, N channel, and N sinks) but i need to use a multiplexing channel processor to route the events(categories) to the proper sink.

      What i am thinking is if i specify the category name in header in Event, HDFSEventSink (and its helper classes) can create a bucket based on header value (and then escape-sequence needs to be honored).

        Activity

        Mubarak Seyed created issue -
        Mubarak Seyed made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Won't Fix [ 2 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Mubarak Seyed
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development