Chukwa
  1. Chukwa
  2. CHUKWA-686

Creating a new adaptor: FileTailingAdaptorPreserveLines

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Data Collection
    • Labels:
      None

      Description

      FileTailingAdaptor is great to tail log files and send them to Hadoop.
      However, last line of the chunk is usually cut which leads to some errors.

      CharFileTailingAdaptorUTF8 can solve such problem.
      Nonetheless, this adaptor calls the MapProcessor.process() method for every
      line in each chunk, thus slowing a lot the Demux phase.

      FileTailingAdaptorPreserveLines adaptor mixes the benefits of the two
      adaptors: the (Demux) speed of FileTailingAdaptor and
      the preservation of lines from CharFileTailingAdaptorUTF8.

      1. chukwa-686.patch
        8 kB
        Sourygna Luangsay
      2. chukwa-686-2.patch
        8 kB
        Sourygna Luangsay

        Activity

        Sourygna Luangsay created issue -
        Sourygna Luangsay made changes -
        Field Original Value New Value
        Description FileTailingAdaptor is great to tail log files and send them to Hadoop.
        However, last line of the chunk is usually cut which leads to some errors.

        CharFileTailingAdaptorUTF8 can solve such problem.
        Nonetheless, this adaptor calls the MapProcessor.process() method for every
        line in each chunk, thus slowing a lot the Demux phase.

        This adaptor that mix the benefits of the two
        adaptors: the (Demux) speed of FileTailingAdaptor and
        the preservation of lines from CharFileTailingAdaptorUTF8.
        Sourygna Luangsay made changes -
        Attachment chukwa-686.patch [ 12579731 ]
        Sourygna Luangsay made changes -
        Description FileTailingAdaptor is great to tail log files and send them to Hadoop.
        However, last line of the chunk is usually cut which leads to some errors.

        CharFileTailingAdaptorUTF8 can solve such problem.
        Nonetheless, this adaptor calls the MapProcessor.process() method for every
        line in each chunk, thus slowing a lot the Demux phase.

        This adaptor that mix the benefits of the two
        adaptors: the (Demux) speed of FileTailingAdaptor and
        the preservation of lines from CharFileTailingAdaptorUTF8.
        FileTailingAdaptor is great to tail log files and send them to Hadoop.
        However, last line of the chunk is usually cut which leads to some errors.

        CharFileTailingAdaptorUTF8 can solve such problem.
        Nonetheless, this adaptor calls the MapProcessor.process() method for every
        line in each chunk, thus slowing a lot the Demux phase.

        FileTailingAdaptorPreserveLines adaptor mixes the benefits of the two
        adaptors: the (Demux) speed of FileTailingAdaptor and
        the preservation of lines from CharFileTailingAdaptorUTF8.
        Hide
        Eric Yang added a comment -

        Can you generate another patch without white space indent changes? It would be good to retain existing indentation to follow Chukwa coding style. (Sun's code conventions except indentation is 2 spaces, not 4)

        The rest of the patch looks fine. Thanks

        Show
        Eric Yang added a comment - Can you generate another patch without white space indent changes? It would be good to retain existing indentation to follow Chukwa coding style. (Sun's code conventions except indentation is 2 spaces, not 4) The rest of the patch looks fine. Thanks
        Hide
        Sourygna Luangsay added a comment -

        Sorry, just changed my Eclipse settings.

        Show
        Sourygna Luangsay added a comment - Sorry, just changed my Eclipse settings.
        Sourygna Luangsay made changes -
        Attachment chukwa-686-2.patch [ 12579740 ]
        Hide
        Eric Yang added a comment -

        +1 looks good.

        Show
        Eric Yang added a comment - +1 looks good.
        Eric Yang made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Eric Yang added a comment -

        I just committed this, thanks Sourygna.

        Show
        Eric Yang added a comment - I just committed this, thanks Sourygna.
        Eric Yang made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Chukwa-trunk #465 (See https://builds.apache.org/job/Chukwa-trunk/465/)
        CHUKWA-686. Added FileTailingAdaptorPreserveLines adaptor. (Sourygna Luangsay via Eric Yang) (Revision 1500353)

        Result = FAILURE
        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1500353
        Files :

        • /incubator/chukwa/trunk/CHANGES.txt
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/FileTailingAdaptorPreserveLines.java
        • /incubator/chukwa/trunk/src/site/apt/agent.apt
        • /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/TestFileTailingAdaptorPreserveLines.java
        Show
        Hudson added a comment - Integrated in Chukwa-trunk #465 (See https://builds.apache.org/job/Chukwa-trunk/465/ ) CHUKWA-686 . Added FileTailingAdaptorPreserveLines adaptor. (Sourygna Luangsay via Eric Yang) (Revision 1500353) Result = FAILURE eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1500353 Files : /incubator/chukwa/trunk/CHANGES.txt /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/FileTailingAdaptorPreserveLines.java /incubator/chukwa/trunk/src/site/apt/agent.apt /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/TestFileTailingAdaptorPreserveLines.java
        Hide
        Hudson added a comment -

        Integrated in Chukwa-trunk #466 (See https://builds.apache.org/job/Chukwa-trunk/466/)
        CHUKWA-686. Added FileTailingAdaptorPreserveLines adaptor. (Sourygna Luangsay via Eric Yang) (Revision 1500367)

        Result = FAILURE
        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1500367
        Files :

        • /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/TestFileTailingAdaptorPreserveLines.java
        Show
        Hudson added a comment - Integrated in Chukwa-trunk #466 (See https://builds.apache.org/job/Chukwa-trunk/466/ ) CHUKWA-686 . Added FileTailingAdaptorPreserveLines adaptor. (Sourygna Luangsay via Eric Yang) (Revision 1500367) Result = FAILURE eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1500367 Files : /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/TestFileTailingAdaptorPreserveLines.java
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        76d 11h 52m 1 Eric Yang 07/Jul/13 02:18
        Patch Available Patch Available Resolved Resolved
        1m 30s 1 Eric Yang 07/Jul/13 02:19

          People

          • Assignee:
            Sourygna Luangsay
            Reporter:
            Sourygna Luangsay
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development