Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3144

Improve Log4jAppender's performance by allowing logging collection of messages

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.7.0
    • Fix Version/s: 1.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently it's only possible to send events one by one with the Log4j appender. (Except if the events are wrapped in an avro record but it's quite cumbersome and might need special handling on the receiving side)

      As the Log4j methods can handle any Object I'd suggest to improve the Log4j appender to treat Collection event as a special case and send its content to Flume with one rpcClient.appendBatch() call.

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #307 (See https://builds.apache.org/job/Flume-trunk-hbase-1/307/)
          FLUME-3144. Improve Log4jAppender's performance by allowing logging (denes: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=66327aa20207ca62cd7c7a4cf9e24724b5ee0dfc)

          • (edit) flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppenderWithAvro.java
          • (edit) flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java
          • (edit) flume-ng-clients/flume-ng-log4jappender/pom.xml
          • (edit) flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in Jenkins build Flume-trunk-hbase-1 #307 (See https://builds.apache.org/job/Flume-trunk-hbase-1/307/ ) FLUME-3144 . Improve Log4jAppender's performance by allowing logging (denes: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=66327aa20207ca62cd7c7a4cf9e24724b5ee0dfc ) (edit) flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppenderWithAvro.java (edit) flume-ng-clients/flume-ng-log4jappender/src/main/java/org/apache/flume/clients/log4jappender/Log4jAppender.java (edit) flume-ng-clients/flume-ng-log4jappender/pom.xml (edit) flume-ng-clients/flume-ng-log4jappender/src/test/java/org/apache/flume/clients/log4jappender/TestLog4jAppender.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flume/pull/151

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flume/pull/151
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 66327aa20207ca62cd7c7a4cf9e24724b5ee0dfc in flume's branch refs/heads/trunk from Denes Arvay
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=66327aa ]

          FLUME-3144. Improve Log4jAppender's performance by allowing logging collection of messages

          Log4jAppender treats Collection messages as a special case making it possible to log
          Collection of events in one Log4j log call. The appender sends these events to the
          receiving Flume instance as one batch with the rpcClient.appendBatch() method.

          This closes #151

          Reviewers: Ferenc Szabo, Miklos Csanady

          (Denes Arvay via Denes Arvay)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 66327aa20207ca62cd7c7a4cf9e24724b5ee0dfc in flume's branch refs/heads/trunk from Denes Arvay [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=66327aa ] FLUME-3144 . Improve Log4jAppender's performance by allowing logging collection of messages Log4jAppender treats Collection messages as a special case making it possible to log Collection of events in one Log4j log call. The appender sends these events to the receiving Flume instance as one batch with the rpcClient.appendBatch() method. This closes #151 Reviewers: Ferenc Szabo, Miklos Csanady (Denes Arvay via Denes Arvay)
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user adenes opened a pull request:

          https://github.com/apache/flume/pull/151

          FLUME-3144. Improve Log4jAppender's performance by allowing logging collection of messages

          `Log4jAppender` treats `Collection` messages as a special case making it possible to log collection of events in one Log4j log call. The appender sends these events to the receiving Flume instance as one batch with the `rpcClient.appendBatch()` method.

          New tests added:

          • logging of a collection of multiple strings, expect one `appendBatch()` call.
          • logging one string, expect one `append()` call.
          • logging of a singleton collection of a string, expect one `append()` call.
          • logging of an empty collection, expect that neither `append()` nor `appendBatch()` was called.
          • logging of a `String` and an arbitrary object in one collection with `AvroReflectionEnabled` set to `true`
          • logging of a `String` and an arbitrary object in one collection with `AvroReflectionEnabled` set to `false`

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/adenes/flume FLUME-3144

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flume/pull/151.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #151


          commit ba4a9a9793104be715d451cc06c42ac73a77e00e
          Author: Denes Arvay <denes@apache.org>
          Date: 2017-08-11T14:22:04Z

          FLUME-3144. Improve Log4jAppender's performance by allowing logging collection of messages

          Log4jAppender treats Collection messages as a special case making it possible to log
          Collection of events in one Log4j log call. The appender sends these events to the
          receiving Flume instance as one batch with the rpcClient.appendBatch() method.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user adenes opened a pull request: https://github.com/apache/flume/pull/151 FLUME-3144 . Improve Log4jAppender's performance by allowing logging collection of messages `Log4jAppender` treats `Collection` messages as a special case making it possible to log collection of events in one Log4j log call. The appender sends these events to the receiving Flume instance as one batch with the `rpcClient.appendBatch()` method. New tests added: logging of a collection of multiple strings, expect one `appendBatch()` call. logging one string, expect one `append()` call. logging of a singleton collection of a string, expect one `append()` call. logging of an empty collection, expect that neither `append()` nor `appendBatch()` was called. logging of a `String` and an arbitrary object in one collection with `AvroReflectionEnabled` set to `true` logging of a `String` and an arbitrary object in one collection with `AvroReflectionEnabled` set to `false` You can merge this pull request into a Git repository by running: $ git pull https://github.com/adenes/flume FLUME-3144 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flume/pull/151.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #151 commit ba4a9a9793104be715d451cc06c42ac73a77e00e Author: Denes Arvay <denes@apache.org> Date: 2017-08-11T14:22:04Z FLUME-3144 . Improve Log4jAppender's performance by allowing logging collection of messages Log4jAppender treats Collection messages as a special case making it possible to log Collection of events in one Log4j log call. The appender sends these events to the receiving Flume instance as one batch with the rpcClient.appendBatch() method.

            People

            • Assignee:
              denes Denes Arvay
              Reporter:
              denes Denes Arvay
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development