Chukwa
  1. Chukwa
  2. CHUKWA-686

Creating a new adaptor: FileTailingAdaptorPreserveLines

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Data Collection
    • Labels:
      None

      Description

      FileTailingAdaptor is great to tail log files and send them to Hadoop.
      However, last line of the chunk is usually cut which leads to some errors.

      CharFileTailingAdaptorUTF8 can solve such problem.
      Nonetheless, this adaptor calls the MapProcessor.process() method for every
      line in each chunk, thus slowing a lot the Demux phase.

      FileTailingAdaptorPreserveLines adaptor mixes the benefits of the two
      adaptors: the (Demux) speed of FileTailingAdaptor and
      the preservation of lines from CharFileTailingAdaptorUTF8.

      1. chukwa-686.patch
        8 kB
        Sourygna Luangsay
      2. chukwa-686-2.patch
        8 kB
        Sourygna Luangsay

        Activity

        Sourygna Luangsay created issue -
        Sourygna Luangsay made changes -
        Field Original Value New Value
        Description FileTailingAdaptor is great to tail log files and send them to Hadoop.
        However, last line of the chunk is usually cut which leads to some errors.

        CharFileTailingAdaptorUTF8 can solve such problem.
        Nonetheless, this adaptor calls the MapProcessor.process() method for every
        line in each chunk, thus slowing a lot the Demux phase.

        This adaptor that mix the benefits of the two
        adaptors: the (Demux) speed of FileTailingAdaptor and
        the preservation of lines from CharFileTailingAdaptorUTF8.
        Sourygna Luangsay made changes -
        Attachment chukwa-686.patch [ 12579731 ]
        Sourygna Luangsay made changes -
        Description FileTailingAdaptor is great to tail log files and send them to Hadoop.
        However, last line of the chunk is usually cut which leads to some errors.

        CharFileTailingAdaptorUTF8 can solve such problem.
        Nonetheless, this adaptor calls the MapProcessor.process() method for every
        line in each chunk, thus slowing a lot the Demux phase.

        This adaptor that mix the benefits of the two
        adaptors: the (Demux) speed of FileTailingAdaptor and
        the preservation of lines from CharFileTailingAdaptorUTF8.
        FileTailingAdaptor is great to tail log files and send them to Hadoop.
        However, last line of the chunk is usually cut which leads to some errors.

        CharFileTailingAdaptorUTF8 can solve such problem.
        Nonetheless, this adaptor calls the MapProcessor.process() method for every
        line in each chunk, thus slowing a lot the Demux phase.

        FileTailingAdaptorPreserveLines adaptor mixes the benefits of the two
        adaptors: the (Demux) speed of FileTailingAdaptor and
        the preservation of lines from CharFileTailingAdaptorUTF8.
        Sourygna Luangsay made changes -
        Attachment chukwa-686-2.patch [ 12579740 ]
        Eric Yang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Eric Yang made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Sourygna Luangsay
            Reporter:
            Sourygna Luangsay
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development