Flume
  1. Flume
  2. FLUME-720

CollectorSink doesn't pass the new format parameter

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v0.9.5
    • Fix Version/s: v0.9.5
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      CollectorSink doesn't properly pass the format parameter down to the EscapedCustomDfs sink.
      For example, this is working fine:
      collectorSource(54001) | escapedCustomDfs("hdfs://hadoop1-m1:8020/", "test", seqfile("SnappyCodec") );

      However, this is using the codec defined in flume-conf.xml
      collectorSource(54001) | collectorSink("hdfs://hadoop1-m1:8020/", "test-", 600000, seqfile("SnappyCodec") );

      By itself this bug would not be very serious, however the problem is that escapedCustomDfs/customDfs are using the same compressor, and they apply it on the whole file, in addition to the compression done natively by the sequence file - this makes the sequence file double compressed and invalid.
      As far as I can tell, the only way to get a valid compressed sequence file is by setting flume.collector.dfs.compress.codec to "None" in flume-site.xml and use the format parameter to specify which compression to use for the sequence file, except that doesn't work...

        Activity

        Jonathan Hsieh made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Jonathan Hsieh made changes -
        Assignee Jonathan Hsieh [ jmhsieh ]
        Jonathan Hsieh made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Fix Version/s v0.9.5 [ 12317558 ]
        Jonathan Hsieh made changes -
        Field Original Value New Value
        Attachment 0001-FLUME-720-CollectorSink-doesn-t-pass-the-new-format-.patch [ 12489374 ]
        Eran Kutner created issue -

          People

          • Assignee:
            Jonathan Hsieh
            Reporter:
            Eran Kutner
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development