Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2246

event body data size can make it configurable for logger sinker

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.6.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Currently logger sinker will dump only 16 bytes of data into log file, better to make it configurable.

      1. FLUME-2246-0.patch
        5 kB
        Ashish Paliwal
      2. FLUME-2246-1.patch
        4 kB
        Ashish Paliwal
      3. FLUME-2246-2.patch
        5 kB
        Ashish Paliwal

        Issue Links

          Activity

          Hide
          wanbin wanbin added a comment -

          I met this problem on version 1.6.0 also. It seem to can't access maxBytesToLog argument.
          I looked over relevant code, I found SinkConfiguration has a problem. I fixed it by modification LOGGER("org.apache.flume.conf.sink.LoggerConfiguration") in SinkConfigurationType enum.

          Show
          wanbin wanbin added a comment - I met this problem on version 1.6.0 also. It seem to can't access maxBytesToLog argument. I looked over relevant code, I found SinkConfiguration has a problem. I fixed it by modification LOGGER("org.apache.flume.conf.sink.LoggerConfiguration") in SinkConfigurationType enum.
          Hide
          ddatsh ddatsh added a comment -

          version1.6.0

          example.conf
          a1.sources = r1
          a1.sinks = k1
          a1.channels = c1
           
          # Describe/configure the source
          a1.sources.r1.type = netcat
          a1.sources.r1.bind = 0.0.0.0
          a1.sources.r1.port = 4444
          # Describe the sink
          a1.sinks.k1.type = logger
          a1.sinks.k1.maxBytesToLog=1024
          
          # Use a channel which buffers events in memory
          a1.channels.c1.type = memory
          a1.channels.c1.capacity = 1000
          a1.channels.c1.transactionCapacity = 100
          a1.channels.c1.maxBytesToLog = 100
           
          # Bind the source and sink to the channel
          a1.sources.r1.channels = c1
          a1.sinks.k1.channel = c1
          

          and telnet to 4444
          paste 1234567890abcdefghijklmn, enter
          console only
          (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 31 32 33 34 35 36 37 38 39 30 61 62 63 64 65 66 1234567890abcdef }

          Show
          ddatsh ddatsh added a comment - version1.6.0 example.conf a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = 0.0.0.0 a1.sources.r1.port = 4444 # Describe the sink a1.sinks.k1.type = logger a1.sinks.k1.maxBytesToLog=1024 # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 a1.channels.c1.maxBytesToLog = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1 and telnet to 4444 paste 1234567890abcdefghijklmn, enter console only (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{} body: 31 32 33 34 35 36 37 38 39 30 61 62 63 64 65 66 1234567890abcdef }
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Flume-trunk-hbase-98 #56 (See https://builds.apache.org/job/Flume-trunk-hbase-98/56/)
          FLUME-2246. Make event data size configurable for logger sinker (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ac999bebeaa59659960fe9e998a0c20b1d476564)

          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java
          • flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Flume-trunk-hbase-98 #56 (See https://builds.apache.org/job/Flume-trunk-hbase-98/56/ ) FLUME-2246 . Make event data size configurable for logger sinker (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ac999bebeaa59659960fe9e998a0c20b1d476564 ) flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in flume-trunk #699 (See https://builds.apache.org/job/flume-trunk/699/)
          FLUME-2246. Make event data size configurable for logger sinker (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ac999bebeaa59659960fe9e998a0c20b1d476564)

          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          • flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java
          • flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in flume-trunk #699 (See https://builds.apache.org/job/flume-trunk/699/ ) FLUME-2246 . Make event data size configurable for logger sinker (roshan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=ac999bebeaa59659960fe9e998a0c20b1d476564 ) flume-ng-doc/sphinx/FlumeUserGuide.rst flume-ng-core/src/main/java/org/apache/flume/sink/LoggerSink.java flume-ng-core/src/test/java/org/apache/flume/sink/TestLoggerSink.java
          Hide
          roshan_naik Roshan Naik added a comment -

          Committed.

          Thanks Ashish Paliwal for the patch.

          Show
          roshan_naik Roshan Naik added a comment - Committed. Thanks Ashish Paliwal for the patch.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ac999bebeaa59659960fe9e998a0c20b1d476564 in flume's branch refs/heads/trunk from Roshan Naik
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ac999be ]

          FLUME-2246. Make event data size configurable for logger sinker

          (Ashish Paliwal via Roshan Naik)

          Show
          jira-bot ASF subversion and git services added a comment - Commit ac999bebeaa59659960fe9e998a0c20b1d476564 in flume's branch refs/heads/trunk from Roshan Naik [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=ac999be ] FLUME-2246 . Make event data size configurable for logger sinker (Ashish Paliwal via Roshan Naik)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 4e50b16abed9dfb03b8b870083a05d3320bcd38e in flume's branch refs/heads/flume-1.6 from Roshan Naik
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4e50b16 ]

          FLUME-2246. Make event data size configurable for logger sinker

          (Ashish Paliwal via Roshan Naik)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 4e50b16abed9dfb03b8b870083a05d3320bcd38e in flume's branch refs/heads/flume-1.6 from Roshan Naik [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4e50b16 ] FLUME-2246 . Make event data size configurable for logger sinker (Ashish Paliwal via Roshan Naik)
          Hide
          roshan_naik Roshan Naik added a comment -

          +1 .. but I will rename "maxBytesToDump" in the test to "maxBytesToLog" before commit.

          Show
          roshan_naik Roshan Naik added a comment - +1 .. but I will rename "maxBytesToDump" in the test to "maxBytesToLog" before commit.
          Hide
          paliwalashish Ashish Paliwal added a comment -

          Updated as per review comments from Roshan

          Show
          paliwalashish Ashish Paliwal added a comment - Updated as per review comments from Roshan
          Hide
          paliwalashish Ashish Paliwal added a comment -

          Updated patch with review comments, minus the assert comment. Not sure how to add assert by intercepting logged data. Verified manually in test logs

          Show
          paliwalashish Ashish Paliwal added a comment - Updated patch with review comments, minus the assert comment. Not sure how to add assert by intercepting logged data. Verified manually in test logs
          Hide
          gopinathan.av Gopinathan A added a comment -

          Thanks for patch Ashish Paliwal

          Show
          gopinathan.av Gopinathan A added a comment - Thanks for patch Ashish Paliwal
          Hide
          paliwalashish Ashish Paliwal added a comment -

          This is easily doable, Event.dumpEvent(Event event, int maxBytes) API needs to be used. As of now LoggerSink doesn't have any configuration.

          Show
          paliwalashish Ashish Paliwal added a comment - This is easily doable, Event.dumpEvent(Event event, int maxBytes) API needs to be used. As of now LoggerSink doesn't have any configuration.

            People

            • Assignee:
              paliwalashish Ashish Paliwal
              Reporter:
              gopinathan.av Gopinathan A
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development