Chukwa
  1. Chukwa
  2. CHUKWA-686

Creating a new adaptor: FileTailingAdaptorPreserveLines

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.6.0
    • Component/s: Data Collection
    • Labels:
      None

      Description

      FileTailingAdaptor is great to tail log files and send them to Hadoop.
      However, last line of the chunk is usually cut which leads to some errors.

      CharFileTailingAdaptorUTF8 can solve such problem.
      Nonetheless, this adaptor calls the MapProcessor.process() method for every
      line in each chunk, thus slowing a lot the Demux phase.

      FileTailingAdaptorPreserveLines adaptor mixes the benefits of the two
      adaptors: the (Demux) speed of FileTailingAdaptor and
      the preservation of lines from CharFileTailingAdaptorUTF8.

      1. chukwa-686-2.patch
        8 kB
        Sourygna Luangsay
      2. chukwa-686.patch
        8 kB
        Sourygna Luangsay

        Activity

        Hide
        Hudson added a comment -

        Integrated in Chukwa-trunk #466 (See https://builds.apache.org/job/Chukwa-trunk/466/)
        CHUKWA-686. Added FileTailingAdaptorPreserveLines adaptor. (Sourygna Luangsay via Eric Yang) (Revision 1500367)

        Result = FAILURE
        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1500367
        Files :

        • /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/TestFileTailingAdaptorPreserveLines.java
        Show
        Hudson added a comment - Integrated in Chukwa-trunk #466 (See https://builds.apache.org/job/Chukwa-trunk/466/ ) CHUKWA-686 . Added FileTailingAdaptorPreserveLines adaptor. (Sourygna Luangsay via Eric Yang) (Revision 1500367) Result = FAILURE eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1500367 Files : /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/TestFileTailingAdaptorPreserveLines.java
        Hide
        Hudson added a comment -

        Integrated in Chukwa-trunk #465 (See https://builds.apache.org/job/Chukwa-trunk/465/)
        CHUKWA-686. Added FileTailingAdaptorPreserveLines adaptor. (Sourygna Luangsay via Eric Yang) (Revision 1500353)

        Result = FAILURE
        eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1500353
        Files :

        • /incubator/chukwa/trunk/CHANGES.txt
        • /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/FileTailingAdaptorPreserveLines.java
        • /incubator/chukwa/trunk/src/site/apt/agent.apt
        • /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/TestFileTailingAdaptorPreserveLines.java
        Show
        Hudson added a comment - Integrated in Chukwa-trunk #465 (See https://builds.apache.org/job/Chukwa-trunk/465/ ) CHUKWA-686 . Added FileTailingAdaptorPreserveLines adaptor. (Sourygna Luangsay via Eric Yang) (Revision 1500353) Result = FAILURE eyang : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1500353 Files : /incubator/chukwa/trunk/CHANGES.txt /incubator/chukwa/trunk/src/main/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/FileTailingAdaptorPreserveLines.java /incubator/chukwa/trunk/src/site/apt/agent.apt /incubator/chukwa/trunk/src/test/java/org/apache/hadoop/chukwa/datacollection/adaptor/filetailer/TestFileTailingAdaptorPreserveLines.java
        Hide
        Eric Yang added a comment -

        I just committed this, thanks Sourygna.

        Show
        Eric Yang added a comment - I just committed this, thanks Sourygna.
        Hide
        Eric Yang added a comment -

        +1 looks good.

        Show
        Eric Yang added a comment - +1 looks good.
        Hide
        Sourygna Luangsay added a comment -

        Sorry, just changed my Eclipse settings.

        Show
        Sourygna Luangsay added a comment - Sorry, just changed my Eclipse settings.
        Hide
        Eric Yang added a comment -

        Can you generate another patch without white space indent changes? It would be good to retain existing indentation to follow Chukwa coding style. (Sun's code conventions except indentation is 2 spaces, not 4)

        The rest of the patch looks fine. Thanks

        Show
        Eric Yang added a comment - Can you generate another patch without white space indent changes? It would be good to retain existing indentation to follow Chukwa coding style. (Sun's code conventions except indentation is 2 spaces, not 4) The rest of the patch looks fine. Thanks

          People

          • Assignee:
            Sourygna Luangsay
            Reporter:
            Sourygna Luangsay
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development